[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:27 dist-test-slave-9b3m krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:27 dist-test-slave-9b3m krb5kdc[399](info): set up 2 sockets
Jun 20 12:19:27 dist-test-slave-9b3m krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 12:19:28.945555 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:29.068266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44507
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:29.175431 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:29.175668 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:29.175705 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:29.179585 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:29.179632 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:29.179656 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:29.179672 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:29.179713 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:29.184782 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:29.185720 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:29.186694 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:29.195681 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:29.195809 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:29.195868 412 server_base.cc:1034] running on GCE node
W20260620 12:19:29.195812 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:29.196481 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:29.197647 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:29.198839 412 hybrid_clock.cc:648] HybridClock initialized: now 1781957969198775 us; error 76 us; skew 500 ppm
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:29.201642 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:29.202472 412 webserver.cc:460] Webserver started at http://127.0.98.62:37887/ using document root <none> and password file <none>
I20260620 12:19:29.202980 412 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:29.203037 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:29.203258 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:29.205238 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "0c258da767e147fbaf60e0e66edc5af1"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "cac5335c257bf24bdf57537be49186b7"
server_key_iv: "ad327968f9e2adfd633e52d58d14af70"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.205739 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c258da767e147fbaf60e0e66edc5af1"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "cac5335c257bf24bdf57537be49186b7"
server_key_iv: "ad327968f9e2adfd633e52d58d14af70"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.209331 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:19:29.211628 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.212859 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:29.212983 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "0c258da767e147fbaf60e0e66edc5af1"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "cac5335c257bf24bdf57537be49186b7"
server_key_iv: "ad327968f9e2adfd633e52d58d14af70"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.213151 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:29.227336 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:29.230618 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:29.230835 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:29.238319 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44507
I20260620 12:19:29.238346 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44507 every 8 connection(s)
I20260620 12:19:29.239249 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:29.242033 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:29.244165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 412
I20260620 12:19:29.244309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:29.245558 392 external_mini_cluster.cc:1428] Setting key e0ef19760f51d861f57d7951cebbac9d
I20260620 12:19:29.247778 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1: Bootstrap starting.
I20260620 12:19:29.250504 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:29.251269 480 log.cc:824] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:29.253029 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1: No bootstrap required, opened a new log
I20260620 12:19:29.256424 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c258da767e147fbaf60e0e66edc5af1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44507 } }
I20260620 12:19:29.256626 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:29.256686 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c258da767e147fbaf60e0e66edc5af1, State: Initialized, Role: FOLLOWER
I20260620 12:19:29.257148 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c258da767e147fbaf60e0e66edc5af1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44507 } }
I20260620 12:19:29.257289 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:29.257354 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:29.257464 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:29.258599 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c258da767e147fbaf60e0e66edc5af1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44507 } }
I20260620 12:19:29.258955 480 leader_election.cc:302] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c258da767e147fbaf60e0e66edc5af1; no voters:
I20260620 12:19:29.259275 480 leader_election.cc:288] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:29.259372 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:29.259579 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [term 1 LEADER]: Becoming Leader. State: Replica: 0c258da767e147fbaf60e0e66edc5af1, State: Running, Role: LEADER
I20260620 12:19:29.259894 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c258da767e147fbaf60e0e66edc5af1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44507 } }
I20260620 12:19:29.260536 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:29.261132 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c258da767e147fbaf60e0e66edc5af1. Latest consensus state: current_term: 1 leader_uuid: "0c258da767e147fbaf60e0e66edc5af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c258da767e147fbaf60e0e66edc5af1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44507 } } }
I20260620 12:19:29.261253 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:29.261581 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c258da767e147fbaf60e0e66edc5af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c258da767e147fbaf60e0e66edc5af1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44507 } } }
I20260620 12:19:29.261667 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:29.261932 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:29.265533 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:29.270939 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:29.248670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.0.98.62:44507)
0620 12:19:29.249134 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:29.249150 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:29.250165 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 12:19:29.253410 (+ 3245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:29.253434 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:29.253741 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:29.254061 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:29.256158 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.257452 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.259165 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.259912 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.260072 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:29.265102 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:29.265114 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:29.265126 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:29.265152 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:29.267192 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:29.267843 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:29.267850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:29.267856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:29.267994 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:29.268384 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:29.268388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:29.268390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:29.268891 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:29.269106 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:29.269763 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 12:19:29.270018 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":142,"threads_started":1}
I20260620 12:19:29.272841 490 catalog_manager.cc:1347] Generated new cluster ID: f9922441f53b4a0bb45903e58f61e905
I20260620 12:19:29.272893 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:29.306878 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:29.307857 490 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:29.313402 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c258da767e147fbaf60e0e66edc5af1: Generated new TSK 0
I20260620 12:19:29.314165 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:29.343976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44507
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:29.452366 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:29.452653 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:29.452729 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:29.456799 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:29.456879 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:29.456969 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:29.462416 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:29.463495 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:29.464496 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:29.474153 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:29.474205 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:29.474252 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:29.474422 506 server_base.cc:1034] running on GCE node
I20260620 12:19:29.474803 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:29.475379 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:29.476547 506 hybrid_clock.cc:648] HybridClock initialized: now 1781957969476533 us; error 28 us; skew 500 ppm
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:29.479244 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:29.480124 506 webserver.cc:460] Webserver started at http://127.0.98.1:40619/ using document root <none> and password file <none>
I20260620 12:19:29.480671 506 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:29.480731 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:29.480958 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:29.483006 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "ddf6dcf4134a4225a97f95952458f050"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "5b833381c95d988b0d195d8694c045d2"
server_key_iv: "1a92b8cd8270b30c12535fe689fca64b"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.483574 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ddf6dcf4134a4225a97f95952458f050"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "5b833381c95d988b0d195d8694c045d2"
server_key_iv: "1a92b8cd8270b30c12535fe689fca64b"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.487335 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:29.489732 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.490890 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:29.491006 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "ddf6dcf4134a4225a97f95952458f050"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "5b833381c95d988b0d195d8694c045d2"
server_key_iv: "1a92b8cd8270b30c12535fe689fca64b"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.491107 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:29.502645 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:29.506129 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:29.506271 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:29.506755 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:29.507493 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:29.507537 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.507575 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:29.507591 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.516429 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42123
I20260620 12:19:29.516505 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42123 every 8 connection(s)
I20260620 12:19:29.517405 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:29.520207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 506
I20260620 12:19:29.520323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:29.520610 392 external_mini_cluster.cc:1428] Setting key 71a919abe377b2a1273377acbeea6ff8
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:19:29.532550 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:29.519265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36673 (local address 127.0.98.62:44507)
0620 12:19:29.519402 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:29.519405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:29.520343 (+ 938us) server_negotiation.cc:407] Connection header received
0620 12:19:29.521797 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:29.521800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:29.521851 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:29.521974 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:29.523071 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.524003 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.525550 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.526075 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.526217 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:29.528689 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:29.528694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:29.528697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:29.528732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:29.530362 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:29.530918 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:29.530923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:29.530924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:29.531004 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:29.531342 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:29.531346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:29.531348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:29.531593 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:29.531734 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:29.532227 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 12:19:29.532327 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:29.533420 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:29.519719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44507 (local address 127.0.98.1:36673)
0620 12:19:29.520194 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:29.520234 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:29.521599 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:29.522131 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:29.522138 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:29.522470 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:29.522921 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:29.522933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.524131 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.524136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:29.525417 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:29.525427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.526259 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.526265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:29.526454 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:29.527051 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:29.527067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:29.528496 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:29.530486 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:29.530492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:29.530513 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:29.530814 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:29.531097 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:29.531101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:29.531103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:29.531245 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:29.531754 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:29.531759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:29.532006 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 12:19:29.532177 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 12:19:29.532425 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":157,"threads_started":1}
I20260620 12:19:29.534672 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44507
I20260620 12:19:29.534900 635 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:29.535382 635 heartbeater.cc:511] Master 127.0.98.62:44507 requested a full tablet report, sending...
I20260620 12:19:29.536937 444 ts_manager.cc:194] Registered new tserver with Master: ddf6dcf4134a4225a97f95952458f050 (127.0.98.1:42123)
I20260620 12:19:29.538717 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:29.550040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44507
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:29.664461 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:29.664736 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:29.664816 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:29.668960 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:29.669031 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:29.669202 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:29.674551 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:29.675586 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:29.676589 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:29.686297 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:29.686328 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:29.686323 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:29.686362 640 server_base.cc:1034] running on GCE node
I20260620 12:19:29.686973 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:29.687589 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:29.688776 640 hybrid_clock.cc:648] HybridClock initialized: now 1781957969688766 us; error 31 us; skew 500 ppm
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:29.691623 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:29.692484 640 webserver.cc:460] Webserver started at http://127.0.98.2:34257/ using document root <none> and password file <none>
I20260620 12:19:29.693097 640 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:29.693195 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:29.693408 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:29.695500 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "8231ea81f0a64e8199e2d065199b8d7b"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "de870cc129a2fbcde841f80c78b6934c"
server_key_iv: "016ced5c6ac5f13f5cb13a264d33ba43"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.696053 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8231ea81f0a64e8199e2d065199b8d7b"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "de870cc129a2fbcde841f80c78b6934c"
server_key_iv: "016ced5c6ac5f13f5cb13a264d33ba43"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.699566 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:19:29.701902 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.703101 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:29.703244 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "8231ea81f0a64e8199e2d065199b8d7b"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "de870cc129a2fbcde841f80c78b6934c"
server_key_iv: "016ced5c6ac5f13f5cb13a264d33ba43"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.703353 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:29.727661 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:29.731423 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:29.731660 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:29.732270 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:29.733172 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:29.733239 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.733306 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:29.733352 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.743146 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34045
I20260620 12:19:29.743184 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34045 every 8 connection(s)
I20260620 12:19:29.744107 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:29.746542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 640
I20260620 12:19:29.746696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:29.746974 392 external_mini_cluster.cc:1428] Setting key f4ad26eb0388d1e7c26bd226529cb966
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:19:29.758870 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:29.746026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57501 (local address 127.0.98.62:44507)
0620 12:19:29.746165 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:29.746168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:29.747096 (+ 928us) server_negotiation.cc:407] Connection header received
0620 12:19:29.748852 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:29.748854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:29.748898 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:29.748968 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:29.750583 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.751109 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.752332 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.752549 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.752656 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:29.755128 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:29.755145 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:29.755147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:29.755169 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:29.756617 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:29.757210 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:29.757215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:29.757217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:29.757295 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:29.757750 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:29.757754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:29.757755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:29.757931 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:29.758043 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:29.758657 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 12:19:29.758755 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:19:29.760403 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:29.746371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44507 (local address 127.0.98.2:57501)
0620 12:19:29.746919 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:29.746979 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:29.748611 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:29.749217 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:29.749229 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:29.749777 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:29.750441 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:29.750456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.751243 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.751249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:29.752204 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:29.752213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.752698 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.752702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:29.752867 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:29.753511 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:29.753527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:29.754967 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:29.756759 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:29.756767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:29.756784 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:29.757100 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:29.757393 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:29.757396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:29.757398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:29.757646 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:29.758082 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:29.758088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:29.758355 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 12:19:29.758555 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 12:19:29.758932 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":115,"threads_started":1}
I20260620 12:19:29.761673 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44507
I20260620 12:19:29.761948 769 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:29.762526 769 heartbeater.cc:511] Master 127.0.98.62:44507 requested a full tablet report, sending...
I20260620 12:19:29.763557 444 ts_manager.cc:194] Registered new tserver with Master: 8231ea81f0a64e8199e2d065199b8d7b (127.0.98.2:34045)
I20260620 12:19:29.764122 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:19:29.776219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44507
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:29.885520 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:29.885756 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:29.885843 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:29.889770 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:29.889823 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:29.889930 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:29.895162 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:29.896142 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:29.897135 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:29.906244 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:29.906227 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:29.906368 774 server_base.cc:1034] running on GCE node
W20260620 12:19:29.906219 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:29.906889 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:29.907531 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:29.908730 774 hybrid_clock.cc:648] HybridClock initialized: now 1781957969908702 us; error 26 us; skew 500 ppm
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:29.911568 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:29.912498 774 webserver.cc:460] Webserver started at http://127.0.98.3:45489/ using document root <none> and password file <none>
I20260620 12:19:29.913163 774 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:29.913233 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:29.913480 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:29.915582 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "a170785106fd4d1b8ed5f90309c8d984"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "8dde7f2efd1f334906e71097ee0ecc5b"
server_key_iv: "75700155361e9ded4594181407cc31dc"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.916225 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a170785106fd4d1b8ed5f90309c8d984"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "8dde7f2efd1f334906e71097ee0ecc5b"
server_key_iv: "75700155361e9ded4594181407cc31dc"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.920042 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 12:19:29.922499 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.923794 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:29.923934 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "a170785106fd4d1b8ed5f90309c8d984"
format_stamp: "Formatted at 2026-06-20 12:19:29 on dist-test-slave-9b3m"
server_key: "8dde7f2efd1f334906e71097ee0ecc5b"
server_key_iv: "75700155361e9ded4594181407cc31dc"
server_key_version: "encryptionkey@0"
I20260620 12:19:29.924045 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:29.946753 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:29.949779 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:29.949980 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:29.950572 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:29.951428 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:29.951509 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.951607 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:29.951646 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:29.961421 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34513
I20260620 12:19:29.961467 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34513 every 8 connection(s)
I20260620 12:19:29.962430 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:29.972105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 774
I20260620 12:19:29.972206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:29.972491 392 external_mini_cluster.cc:1428] Setting key a7f45504d73519632ccd3abdc424e671
I20260620 12:19:29.976142 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:29.964234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56699 (local address 127.0.98.62:44507)
0620 12:19:29.964377 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:29.964381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:29.965256 (+ 875us) server_negotiation.cc:407] Connection header received
0620 12:19:29.966555 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:29.966558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:29.966614 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:29.966718 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:29.967826 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.968353 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.969555 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.969773 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.969879 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:29.972326 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:29.972330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:29.972334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:29.972368 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:29.974146 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:29.974721 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:29.974728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:29.974733 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:29.974828 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:29.975215 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:29.975219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:29.975220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:29.975422 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:29.975507 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:29.975926 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 12:19:29.976020 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:19:29.976887 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:29.964657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44507 (local address 127.0.98.3:56699)
0620 12:19:29.965108 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:29.965148 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:29.966355 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:29.966879 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:29.966885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:29.967221 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:29.967690 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:29.967703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.968473 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.968477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:29.969436 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:29.969445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:29.969924 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:29.969928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:29.970086 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:29.970665 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:29.970680 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:29.972139 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:29.974262 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:29.974267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:29.974280 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:29.974599 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:29.974928 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:29.974938 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:29.974940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:29.975110 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:29.975511 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:29.975515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:29.975729 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 12:19:29.975876 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 12:19:29.976099 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260620 12:19:29.978001 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44507
I20260620 12:19:29.978217 903 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:29.978653 903 heartbeater.cc:511] Master 127.0.98.62:44507 requested a full tablet report, sending...
I20260620 12:19:29.979585 444 ts_manager.cc:194] Registered new tserver with Master: a170785106fd4d1b8ed5f90309c8d984 (127.0.98.3:34513)
I20260620 12:19:29.980084 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56699
I20260620 12:19:29.986946 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:19:29 dist-test-slave-9b3m krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:19:30 dist-test-slave-9b3m krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957969, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:30.014183 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:30.004650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.0.98.1:42123)
0620 12:19:30.004946 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:30.004960 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:30.004985 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:19:30.005132 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:30.005137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:30.005327 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:30.005450 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:30.006057 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:30.006945 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:30.007876 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:30.008245 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:30.008328 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:30.010837 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:30.010845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:30.010858 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:30.010879 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:30.012346 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:30.012731 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:30.012736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:30.012737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:30.012810 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:30.013174 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:30.013177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:30.013179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:30.013429 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:30.013634 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:30.013774 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 12:19:30.013960 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
W20260620 12:19:30.015177 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57594
Jun 20 12:19:30 dist-test-slave-9b3m krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:19:30 dist-test-slave-9b3m krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:30.042363 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:30.032749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.0.98.1:42123)
0620 12:19:30.032943 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:30.032947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:30.033086 (+ 139us) server_negotiation.cc:407] Connection header received
0620 12:19:30.033269 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:30.033272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:30.033313 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:30.033384 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:30.034143 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:30.034665 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:30.035485 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:30.035717 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:30.035796 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:30.038745 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:30.038748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:30.038766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:30.038797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:30.040332 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:30.040866 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:30.040873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:30.040874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:30.040977 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:30.041354 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:30.041357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:30.041358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:30.041559 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:30.041682 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:30.042001 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 12:19:30.042175 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 12:19:30.044263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 506
I20260620 12:19:30.050493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 640
I20260620 12:19:30.058725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 774
I20260620 12:19:30.064679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 412
2026-06-20T12:19:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2171 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:30 dist-test-slave-9b3m krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:30 dist-test-slave-9b3m krb5kdc[917](info): set up 2 sockets
Jun 20 12:19:30 dist-test-slave-9b3m krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 12:19:31.148823 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:31 dist-test-slave-9b3m krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:31.268318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:31.375104 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:31.375355 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:31.375393 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:31.379372 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:31.379428 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:31.379447 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:31.379464 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:31.379479 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:31.384538 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:31.385579 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:31.386528 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:31.394950 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:31.394999 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:31.394924 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:31.395262 930 server_base.cc:1034] running on GCE node
I20260620 12:19:31.395805 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:31.396921 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:31.398083 930 hybrid_clock.cc:648] HybridClock initialized: now 1781957971398062 us; error 35 us; skew 500 ppm
Jun 20 12:19:31 dist-test-slave-9b3m krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:31.400883 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:31.401835 930 webserver.cc:460] Webserver started at http://127.0.98.62:46333/ using document root <none> and password file <none>
I20260620 12:19:31.402376 930 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:31.402477 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:31.402702 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:31.404644 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "fdac5eda148f4baf954b61d015536393"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "810ba603ca87c64ae68de91cfe6eb858"
server_key_iv: "62ba58ff5a17088ec14561c4ac659ef7"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.405248 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdac5eda148f4baf954b61d015536393"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "810ba603ca87c64ae68de91cfe6eb858"
server_key_iv: "62ba58ff5a17088ec14561c4ac659ef7"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.408720 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 12:19:31.411015 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:31.412153 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:31.412286 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "fdac5eda148f4baf954b61d015536393"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "810ba603ca87c64ae68de91cfe6eb858"
server_key_iv: "62ba58ff5a17088ec14561c4ac659ef7"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.412390 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:31.446933 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:31.450359 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:31.450574 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:31.457544 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37827
I20260620 12:19:31.457569 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37827 every 8 connection(s)
I20260620 12:19:31.458465 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:31.461158 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:31.464300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 930
I20260620 12:19:31.464416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:31.464680 392 external_mini_cluster.cc:1428] Setting key ab218c29e0adec60cca7c336d4449272
I20260620 12:19:31.467033 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393: Bootstrap starting.
I20260620 12:19:31.469220 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:31.469928 998 log.cc:824] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:31.471690 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393: No bootstrap required, opened a new log
Jun 20 12:19:31 dist-test-slave-9b3m krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:31.475858 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdac5eda148f4baf954b61d015536393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37827 } }
I20260620 12:19:31.476073 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:31.476107 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdac5eda148f4baf954b61d015536393, State: Initialized, Role: FOLLOWER
I20260620 12:19:31.476591 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdac5eda148f4baf954b61d015536393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37827 } }
I20260620 12:19:31.476742 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:31.476809 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:31.476969 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:31.477813 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdac5eda148f4baf954b61d015536393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37827 } }
I20260620 12:19:31.478161 998 leader_election.cc:302] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdac5eda148f4baf954b61d015536393; no voters:
I20260620 12:19:31.478472 998 leader_election.cc:288] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:31.478576 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:31.478796 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [term 1 LEADER]: Becoming Leader. State: Replica: fdac5eda148f4baf954b61d015536393, State: Running, Role: LEADER
I20260620 12:19:31.479094 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdac5eda148f4baf954b61d015536393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37827 } }
I20260620 12:19:31.479240 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:31.466211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.0.98.62:37827)
0620 12:19:31.466669 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:31.466685 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:31.466724 (+ 39us) server_negotiation.cc:407] Connection header received
0620 12:19:31.467434 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:31.467453 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:31.467780 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:31.468115 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:31.468907 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.470024 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.470894 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.471379 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.471548 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:31.474377 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:31.474382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:31.474394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:31.474418 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:31.476046 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:31.476506 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:31.476518 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:31.476523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:31.476629 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:31.476930 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:31.476934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:31.476935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:31.477405 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:31.477610 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:31.477886 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 12:19:31.478198 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":136,"threads_started":1}
I20260620 12:19:31.479799 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:31.480243 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdac5eda148f4baf954b61d015536393. Latest consensus state: current_term: 1 leader_uuid: "fdac5eda148f4baf954b61d015536393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdac5eda148f4baf954b61d015536393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37827 } } }
I20260620 12:19:31.480365 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:31.480660 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdac5eda148f4baf954b61d015536393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdac5eda148f4baf954b61d015536393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37827 } } }
I20260620 12:19:31.480762 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:31.481438 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:31.484423 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:31.489912 1007 catalog_manager.cc:1347] Generated new cluster ID: a84aeb47bcdd49838d3f2fdb90d18ff8
I20260620 12:19:31.489977 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:31.500854 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:31.501880 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:31.512418 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdac5eda148f4baf954b61d015536393: Generated new TSK 0
I20260620 12:19:31.513320 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:31.546016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37827
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:31.659438 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:31.659686 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:31.659756 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:31.663739 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:31.663794 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:31.663897 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:31.669003 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:31.670054 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:31.671065 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:31.680598 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:31.680611 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:31.680606 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:31.680855 1024 server_base.cc:1034] running on GCE node
I20260620 12:19:31.681519 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:31.682221 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:31.683435 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781957971683406 us; error 50 us; skew 500 ppm
Jun 20 12:19:31 dist-test-slave-9b3m krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:31.686575 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:31.687485 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41641/ using document root <none> and password file <none>
I20260620 12:19:31.688035 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:31.688095 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:31.688263 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:31.690428 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "800b20327cfb4ff29e73c6fb94a1434b"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "10a9e3322e06ca301b0115b5492d0ba6"
server_key_iv: "da440aa364df1665a813f85c28cd05be"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.690961 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "800b20327cfb4ff29e73c6fb94a1434b"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "10a9e3322e06ca301b0115b5492d0ba6"
server_key_iv: "da440aa364df1665a813f85c28cd05be"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.694895 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:19:31.697340 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:31.698606 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:31.698706 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "800b20327cfb4ff29e73c6fb94a1434b"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "10a9e3322e06ca301b0115b5492d0ba6"
server_key_iv: "da440aa364df1665a813f85c28cd05be"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.698807 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:31.714164 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:31.717298 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:31.717444 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:31.717917 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:31.718626 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:31.718670 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:31.718708 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:31.718722 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:31.727581 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34449
I20260620 12:19:31.727605 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34449 every 8 connection(s)
I20260620 12:19:31.728415 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:31.732219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 1024
I20260620 12:19:31.732372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:31.732682 392 external_mini_cluster.cc:1428] Setting key 3a83c918042ce01a312b3f9f6307218c
Jun 20 12:19:31 dist-test-slave-9b3m krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:19:31.743498 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:31.730209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56845 (local address 127.0.98.62:37827)
0620 12:19:31.730410 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:31.730414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:31.731160 (+ 746us) server_negotiation.cc:407] Connection header received
0620 12:19:31.732535 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:31.732539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:31.732601 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:31.732707 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:31.734101 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.734863 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.736422 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.736810 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.736918 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:31.739571 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:31.739575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:31.739577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:31.739599 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:31.741279 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:31.741801 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:31.741805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:31.741807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:31.741891 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:31.742276 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:31.742279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:31.742281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:31.742478 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:31.742590 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:31.743188 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 12:19:31.743316 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:31.744418 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:31.730589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37827 (local address 127.0.98.1:56845)
0620 12:19:31.731014 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:31.731055 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:31.732279 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:31.732885 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:31.732892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:31.733301 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:31.733944 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:31.733960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.734998 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.735002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:31.736257 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:31.736304 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.736965 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.736969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:31.737172 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:31.737808 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:31.737824 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:31.739396 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:31.741401 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:31.741407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:31.741421 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:31.741684 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:31.742018 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:31.742021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:31.742023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:31.742161 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:31.742601 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:31.742610 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:31.742902 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 12:19:31.743132 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 12:19:31.743471 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260620 12:19:31.745745 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37827
I20260620 12:19:31.746027 1153 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:31.746595 1153 heartbeater.cc:511] Master 127.0.98.62:37827 requested a full tablet report, sending...
I20260620 12:19:31.748363 962 ts_manager.cc:194] Registered new tserver with Master: 800b20327cfb4ff29e73c6fb94a1434b (127.0.98.1:34449)
I20260620 12:19:31.750047 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:31.760574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37827
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:31.865902 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:31.866128 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:31.866169 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:31.869969 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:31.870014 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:31.870069 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:31.874955 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:31.875910 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:31.876801 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:31.885999 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:31.885998 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:31.886184 1158 server_base.cc:1034] running on GCE node
W20260620 12:19:31.885994 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:31.886915 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:31.887578 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:31.888779 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781957971888768 us; error 27 us; skew 500 ppm
Jun 20 12:19:31 dist-test-slave-9b3m krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:31.891624 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:31.892521 1158 webserver.cc:460] Webserver started at http://127.0.98.2:39651/ using document root <none> and password file <none>
I20260620 12:19:31.893149 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:31.893249 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:31.893491 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:31.895535 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "7873b01061524204894ca2edb5ab8f32"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "ef6b59f1cc9efc58ff2b1fa3de6f8d05"
server_key_iv: "34f55acb2be332ada0cadb712396324b"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.896101 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7873b01061524204894ca2edb5ab8f32"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "ef6b59f1cc9efc58ff2b1fa3de6f8d05"
server_key_iv: "34f55acb2be332ada0cadb712396324b"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.899770 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 12:19:31.902201 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:31.903524 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:31.903620 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "7873b01061524204894ca2edb5ab8f32"
format_stamp: "Formatted at 2026-06-20 12:19:31 on dist-test-slave-9b3m"
server_key: "ef6b59f1cc9efc58ff2b1fa3de6f8d05"
server_key_iv: "34f55acb2be332ada0cadb712396324b"
server_key_version: "encryptionkey@0"
I20260620 12:19:31.903698 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:31.922950 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:31.925915 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:31.926061 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:31.926571 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:31.927304 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:31.927358 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:31.927397 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:31.927412 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:31.936354 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38623
I20260620 12:19:31.936375 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38623 every 8 connection(s)
I20260620 12:19:31.937288 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:31.946383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 1158
I20260620 12:19:31.946519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:31.946801 392 external_mini_cluster.cc:1428] Setting key c54173dbe6b4d672d5013589f445a72f
Jun 20 12:19:31 dist-test-slave-9b3m krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:31.951306 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:31.939048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43771 (local address 127.0.98.62:37827)
0620 12:19:31.939182 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:31.939185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:31.940411 (+ 1226us) server_negotiation.cc:407] Connection header received
0620 12:19:31.941358 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:31.941362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:31.941423 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:31.941529 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:31.942647 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.943185 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.944429 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.944834 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.944979 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:31.947762 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:31.947765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:31.947767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:31.947791 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:31.949317 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:31.949841 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:31.949847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:31.949849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:31.949928 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:31.950273 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:31.950276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:31.950278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:31.950449 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:31.950551 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:31.951030 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 12:19:31.951143 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:19:31.952198 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:31.939412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37827 (local address 127.0.98.2:43771)
0620 12:19:31.940269 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:31.940309 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:31.941157 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:31.941695 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:31.941702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:31.942037 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:31.942494 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:31.942506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.943308 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.943312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:31.944259 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:31.944274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:31.945061 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:31.945066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:31.945268 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:31.945899 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:31.945917 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:31.947602 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:31.949470 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:31.949475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:31.949488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:31.949736 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:31.950033 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:31.950036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:31.950038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:31.950171 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:31.950565 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:31.950572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:31.950799 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 12:19:31.950975 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 12:19:31.951269 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":526,"threads_started":1}
I20260620 12:19:31.953365 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37827
I20260620 12:19:31.953617 1287 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:31.954182 1287 heartbeater.cc:511] Master 127.0.98.62:37827 requested a full tablet report, sending...
I20260620 12:19:31.955314 962 ts_manager.cc:194] Registered new tserver with Master: 7873b01061524204894ca2edb5ab8f32 (127.0.98.2:38623)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:19:31.956125 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:19:31.973588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37827
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:32.084456 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:32.084678 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:32.084721 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:32.088652 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:32.088703 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:32.088760 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:32.093799 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:32.094725 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:32.095633 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:32.104907 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:32.104950 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:32.105003 1292 server_base.cc:1034] running on GCE node
W20260620 12:19:32.104951 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:32.105654 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:32.106261 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:32.107455 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781957972107444 us; error 24 us; skew 500 ppm
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:32.110019 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:32.110869 1292 webserver.cc:460] Webserver started at http://127.0.98.3:42427/ using document root <none> and password file <none>
I20260620 12:19:32.111410 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:32.111497 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:32.111711 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:32.113735 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "c1b89da1b3234d85b1c7dd5981986750"
format_stamp: "Formatted at 2026-06-20 12:19:32 on dist-test-slave-9b3m"
server_key: "b1956ed89886d4554bd62e6488ec0adf"
server_key_iv: "0deaab9b0fa49b7be9cd295fc713a080"
server_key_version: "encryptionkey@0"
I20260620 12:19:32.114289 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c1b89da1b3234d85b1c7dd5981986750"
format_stamp: "Formatted at 2026-06-20 12:19:32 on dist-test-slave-9b3m"
server_key: "b1956ed89886d4554bd62e6488ec0adf"
server_key_iv: "0deaab9b0fa49b7be9cd295fc713a080"
server_key_version: "encryptionkey@0"
I20260620 12:19:32.117897 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:32.120254 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:32.121515 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:32.121640 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "c1b89da1b3234d85b1c7dd5981986750"
format_stamp: "Formatted at 2026-06-20 12:19:32 on dist-test-slave-9b3m"
server_key: "b1956ed89886d4554bd62e6488ec0adf"
server_key_iv: "0deaab9b0fa49b7be9cd295fc713a080"
server_key_version: "encryptionkey@0"
I20260620 12:19:32.121742 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:32.146675 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:32.149616 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:32.149803 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:32.150346 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:32.151111 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:32.151180 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:32.151248 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:32.151298 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:32.160108 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36231
I20260620 12:19:32.160135 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36231 every 8 connection(s)
I20260620 12:19:32.160921 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:32.169085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 1292
I20260620 12:19:32.169200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:32.169533 392 external_mini_cluster.cc:1428] Setting key 9bbf44f2b2acfe7f61fc044ea2c620f5
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:32.174499 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.162639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59435 (local address 127.0.98.62:37827)
0620 12:19:32.162776 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.162779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.163546 (+ 767us) server_negotiation.cc:407] Connection header received
0620 12:19:32.164882 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.164885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.164942 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.165044 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:32.166169 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.166693 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.167790 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.168040 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.168151 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.170868 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:32.170873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:32.170878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:32.170908 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:32.172466 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.172985 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.172989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.172990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.173138 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.173473 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.173476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.173478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.173653 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:32.173755 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.174218 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.174326 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:19:32.175202 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.163000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37827 (local address 127.0.98.3:59435)
0620 12:19:32.163402 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:32.163445 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:32.164661 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:32.165226 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:32.165232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:32.165552 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:32.166026 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.166039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.166823 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.166831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.167668 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.167676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.168184 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.168188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.168347 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.168911 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:32.168933 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:32.170677 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:32.172604 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:32.172609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:32.172622 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:32.172869 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:32.173238 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:32.173241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:32.173243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:32.173377 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:32.173754 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:32.173760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:32.173999 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 12:19:32.174178 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 12:19:32.174467 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
I20260620 12:19:32.176196 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37827
I20260620 12:19:32.176447 1421 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:32.176877 1421 heartbeater.cc:511] Master 127.0.98.62:37827 requested a full tablet report, sending...
I20260620 12:19:32.177824 962 ts_manager.cc:194] Registered new tserver with Master: c1b89da1b3234d85b1c7dd5981986750 (127.0.98.3:36231)
I20260620 12:19:32.178356 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59435
I20260620 12:19:32.183043 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:19:32.192634 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.186229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.0.98.62:37827)
0620 12:19:32.186354 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.186358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.186525 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:19:32.186610 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.186612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.186647 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.186708 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:32.187337 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.187815 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.188677 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.188870 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.188942 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.190119 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:32.190122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:32.190124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:32.190143 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:32.191284 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.191643 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.191645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.191646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.191711 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.191942 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.191944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.191947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.192168 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:32.192267 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.192451 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.192532 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:19:32.196830 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:19:32.199846 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:32.212934 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.207700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34449 (local address 127.0.0.1:33252)
0620 12:19:32.208004 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:32.208050 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:32.208167 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:32.208902 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:32.208908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:32.208931 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:32.209140 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.209146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.210516 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.210519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.211524 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.211536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.212392 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.212396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.212504 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.212539 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 12:19:32.212669 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 12:19:32.212794 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260620 12:19:32.212940 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.207812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.0.98.1:34449)
0620 12:19:32.208190 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.208206 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.208232 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:19:32.208311 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.208316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.208527 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.208684 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:32.209260 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.210349 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.211639 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.212264 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.212391 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.212535 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.212697 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.212785 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":101,"threads_started":1}
I20260620 12:19:32.214713 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.208679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38623 (local address 127.0.0.1:53572)
0620 12:19:32.209587 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:32.209600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:32.209689 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:32.210233 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:32.210235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:32.210251 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:32.210329 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.210334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.211680 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.211684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.213156 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.213166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.214318 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.214322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.214420 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.214441 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:19:32.214511 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 12:19:32.214579 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":68,"thread_start_us":81,"threads_started":1}
I20260620 12:19:32.214603 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.208568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.0.98.3:36231)
0620 12:19:32.209360 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.209374 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.209434 (+ 60us) server_negotiation.cc:407] Connection header received
0620 12:19:32.209571 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.209576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.209769 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.209932 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:32.210114 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.211164 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.213191 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.214013 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.214125 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.214252 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.214357 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.214438 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":88,"threads_started":1}
I20260620 12:19:32.215345 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.209061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.0.98.2:38623)
0620 12:19:32.209353 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.209365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.209700 (+ 335us) server_negotiation.cc:407] Connection header received
0620 12:19:32.209788 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.209793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.209961 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.210092 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:32.210460 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.211527 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.213271 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.214035 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.214125 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.214252 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.214966 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.215209 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
I20260620 12:19:32.214850 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.208366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36231 (local address 127.0.0.1:51150)
0620 12:19:32.209357 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:32.209369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:32.209437 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:32.209910 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:32.209913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:32.209930 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:32.210018 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.210024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.211307 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.211310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.213013 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.213026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.214136 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.214140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.214226 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.214248 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:19:32.214331 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 12:19:32.214715 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":93,"thread_start_us":99,"threads_started":1}
I20260620 12:19:32.215996 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b3eabd1c74938aeea8a7f8c606ff8 (DEFAULT_TABLE table=test-table [id=310b63b59cf04447a28f35792de5d1ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:32.217149 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b3eabd1c74938aeea8a7f8c606ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:32.218010 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b3eabd1c74938aeea8a7f8c606ff8 (DEFAULT_TABLE table=test-table [id=310b63b59cf04447a28f35792de5d1ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:32.218266 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b3eabd1c74938aeea8a7f8c606ff8 (DEFAULT_TABLE table=test-table [id=310b63b59cf04447a28f35792de5d1ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:32.218931 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b3eabd1c74938aeea8a7f8c606ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:32.219147 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b3eabd1c74938aeea8a7f8c606ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:32.224247 1438 tablet_bootstrap.cc:492] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32: Bootstrap starting.
I20260620 12:19:32.225008 1439 tablet_bootstrap.cc:492] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b: Bootstrap starting.
I20260620 12:19:32.225682 1440 tablet_bootstrap.cc:492] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750: Bootstrap starting.
I20260620 12:19:32.226822 1438 tablet_bootstrap.cc:654] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:32.227068 1439 tablet_bootstrap.cc:654] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:32.227607 1440 tablet_bootstrap.cc:654] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:32.228117 1439 log.cc:824] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:32.228111 1438 log.cc:824] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:32.228754 1440 log.cc:824] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:32.230057 1438 tablet_bootstrap.cc:492] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32: No bootstrap required, opened a new log
I20260620 12:19:32.230099 1439 tablet_bootstrap.cc:492] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b: No bootstrap required, opened a new log
I20260620 12:19:32.230255 1438 ts_tablet_manager.cc:1397] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:19:32.230259 1439 ts_tablet_manager.cc:1397] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 12:19:32.230484 1440 tablet_bootstrap.cc:492] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750: No bootstrap required, opened a new log
I20260620 12:19:32.230638 1440 ts_tablet_manager.cc:1397] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 12:19:32.232970 1438 raft_consensus.cc:348] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.233263 1438 raft_consensus.cc:374] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:32.233326 1438 raft_consensus.cc:729] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7873b01061524204894ca2edb5ab8f32, State: Initialized, Role: FOLLOWER
I20260620 12:19:32.233765 1438 consensus_queue.cc:260] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.234058 1440 raft_consensus.cc:348] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.234021 1439 raft_consensus.cc:348] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.234264 1440 raft_consensus.cc:374] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:32.234265 1439 raft_consensus.cc:374] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:32.234328 1439 raft_consensus.cc:729] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800b20327cfb4ff29e73c6fb94a1434b, State: Initialized, Role: FOLLOWER
I20260620 12:19:32.234328 1440 raft_consensus.cc:729] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b89da1b3234d85b1c7dd5981986750, State: Initialized, Role: FOLLOWER
I20260620 12:19:32.234447 1287 heartbeater.cc:503] Master 127.0.98.62:37827 was elected leader, sending a full tablet report...
I20260620 12:19:32.234717 1438 ts_tablet_manager.cc:1428] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 12:19:32.234742 1439 consensus_queue.cc:260] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.234742 1440 consensus_queue.cc:260] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.235409 1421 heartbeater.cc:503] Master 127.0.98.62:37827 was elected leader, sending a full tablet report...
I20260620 12:19:32.235692 1153 heartbeater.cc:503] Master 127.0.98.62:37827 was elected leader, sending a full tablet report...
I20260620 12:19:32.235960 1440 ts_tablet_manager.cc:1428] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 12:19:32.236507 1439 ts_tablet_manager.cc:1428] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:19:32.248934 1444 raft_consensus.cc:482] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:32.249135 1444 raft_consensus.cc:504] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.250131 1444 leader_election.cc:288] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 800b20327cfb4ff29e73c6fb94a1434b (127.0.98.1:34449), c1b89da1b3234d85b1c7dd5981986750 (127.0.98.3:36231)
I20260620 12:19:32.254329 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.250520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35467 (local address 127.0.98.1:34449)
0620 12:19:32.250643 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.250648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.250662 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:19:32.250760 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.250763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.250805 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.250901 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:32.251242 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.251755 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.253302 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.253704 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.253844 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.253914 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.254128 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.254200 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:19:32.254329 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.250435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34449 (local address 127.0.98.2:35467)
0620 12:19:32.250538 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:32.250570 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:32.250656 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:32.250941 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:32.250945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:32.250965 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:32.251092 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.251098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.251935 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.251940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.253169 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.253185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.253843 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.253847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.253970 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.253995 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:19:32.254110 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 12:19:32.254200 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":29}
I20260620 12:19:32.254470 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.250800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53469 (local address 127.0.98.3:36231)
0620 12:19:32.250931 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.250935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.251158 (+ 223us) server_negotiation.cc:407] Connection header received
0620 12:19:32.251287 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.251290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.251326 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.251412 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:32.251667 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.252311 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.253464 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.253885 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.254021 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.254080 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.254263 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.254332 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:19:32.254622 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.250743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36231 (local address 127.0.98.2:53469)
0620 12:19:32.251079 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:32.251089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:32.251192 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:32.251434 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:32.251437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:32.251451 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:32.251569 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.251574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.252442 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.252446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.253337 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:32.253345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.254021 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.254024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:32.254124 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.254152 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:19:32.254249 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 12:19:32.254314 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":29,"thread_start_us":121,"threads_started":1}
I20260620 12:19:32.255064 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b3eabd1c74938aeea8a7f8c606ff8" candidate_uuid: "7873b01061524204894ca2edb5ab8f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "800b20327cfb4ff29e73c6fb94a1434b" is_pre_election: true
I20260620 12:19:32.255249 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b3eabd1c74938aeea8a7f8c606ff8" candidate_uuid: "7873b01061524204894ca2edb5ab8f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b89da1b3234d85b1c7dd5981986750" is_pre_election: true
I20260620 12:19:32.255359 1107 raft_consensus.cc:2454] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7873b01061524204894ca2edb5ab8f32 in term 0.
I20260620 12:19:32.255494 1375 raft_consensus.cc:2454] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7873b01061524204894ca2edb5ab8f32 in term 0.
I20260620 12:19:32.255844 1174 leader_election.cc:302] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7873b01061524204894ca2edb5ab8f32, 800b20327cfb4ff29e73c6fb94a1434b; no voters:
I20260620 12:19:32.256130 1444 raft_consensus.cc:2781] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:32.256202 1444 raft_consensus.cc:482] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:32.256232 1444 raft_consensus.cc:3037] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:32.257256 1444 raft_consensus.cc:504] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.257575 1444 leader_election.cc:288] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [CANDIDATE]: Term 1 election: Requested vote from peers 800b20327cfb4ff29e73c6fb94a1434b (127.0.98.1:34449), c1b89da1b3234d85b1c7dd5981986750 (127.0.98.3:36231)
I20260620 12:19:32.258014 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b3eabd1c74938aeea8a7f8c606ff8" candidate_uuid: "7873b01061524204894ca2edb5ab8f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "800b20327cfb4ff29e73c6fb94a1434b"
I20260620 12:19:32.258071 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b3eabd1c74938aeea8a7f8c606ff8" candidate_uuid: "7873b01061524204894ca2edb5ab8f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b89da1b3234d85b1c7dd5981986750"
I20260620 12:19:32.258152 1107 raft_consensus.cc:3037] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:32.258173 1375 raft_consensus.cc:3037] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:32.259461 1375 raft_consensus.cc:2454] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P c1b89da1b3234d85b1c7dd5981986750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7873b01061524204894ca2edb5ab8f32 in term 1.
I20260620 12:19:32.259460 1107 raft_consensus.cc:2454] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 800b20327cfb4ff29e73c6fb94a1434b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7873b01061524204894ca2edb5ab8f32 in term 1.
I20260620 12:19:32.259740 1174 leader_election.cc:302] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7873b01061524204894ca2edb5ab8f32, c1b89da1b3234d85b1c7dd5981986750; no voters:
I20260620 12:19:32.259910 1444 raft_consensus.cc:2781] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:32.260144 1444 raft_consensus.cc:686] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [term 1 LEADER]: Becoming Leader. State: Replica: 7873b01061524204894ca2edb5ab8f32, State: Running, Role: LEADER
I20260620 12:19:32.260465 1444 consensus_queue.cc:237] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } }
I20260620 12:19:32.263579 961 catalog_manager.cc:5495] T 0a6b3eabd1c74938aeea8a7f8c606ff8 P 7873b01061524204894ca2edb5ab8f32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7873b01061524204894ca2edb5ab8f32 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7873b01061524204894ca2edb5ab8f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800b20327cfb4ff29e73c6fb94a1434b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b89da1b3234d85b1c7dd5981986750" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7873b01061524204894ca2edb5ab8f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38623 } health_report { overall_health: HEALTHY } } }
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:32.278873 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.269666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.0.98.1:34449)
0620 12:19:32.269813 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.269818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.269950 (+ 132us) server_negotiation.cc:407] Connection header received
0620 12:19:32.270045 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.270048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.270090 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.270161 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:32.270792 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.271361 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.272529 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.272757 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.272869 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.275295 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:32.275308 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:32.275359 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:32.275383 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:32.277293 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.277648 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.277653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.277654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.277737 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.277997 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.278000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.278001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.278231 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:32.278434 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.278549 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.278639 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957972, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:32.303987 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.296072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.0.98.1:34449)
0620 12:19:32.296331 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.296334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.296349 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:19:32.296387 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.296389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.296429 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.296504 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:32.297219 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.297729 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.298568 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.298795 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.298883 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.301091 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:32.301095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:32.301098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:32.301125 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:32.302544 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.302908 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.302911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.302912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.302987 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.303236 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.303239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.303241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.303437 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:32.303546 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.303770 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.303868 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:32.329913 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:32.321419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.0.98.1:34449)
0620 12:19:32.321567 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:32.321572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:32.321739 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:19:32.321912 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:32.321915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:32.321970 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:32.322063 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:32.322775 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.323323 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.324231 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:32.324504 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:32.324628 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:32.327227 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:32.327231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:32.327233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:32.327257 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:32.328526 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.328885 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.328888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.328889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.328964 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:32.329245 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:32.329247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:32.329248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:32.329405 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:32.329508 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:32.329681 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 12:19:32.329771 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 12:19:32.331260 1067 tablet_service.cc:2908] Rejecting scan request for tablet 0a6b3eabd1c74938aeea8a7f8c606ff8: Uninitialized: clean time has not yet been initialized
I20260620 12:19:32.333319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 1024
I20260620 12:19:32.339227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 1158
I20260620 12:19:32.344695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 1292
I20260620 12:19:32.349807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 930
2026-06-20T12:19:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2285 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[1462](info): set up 2 sockets
Jun 20 12:19:32 dist-test-slave-9b3m krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260620 12:19:33.458060 392 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 12:19:33.603672 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.145s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:33 dist-test-slave-9b3m krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:33.716527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:33.824832 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:33.825155 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:33.825263 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:33.829283 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:33.829360 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:19:33.829391 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:33.829432 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:33.829479 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:33.829517 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:33.834743 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34727
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:33.835776 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:33.836786 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:33.845202 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:33.845391 1475 server_base.cc:1034] running on GCE node
W20260620 12:19:33.845242 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:33.845275 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:33.846047 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:33.847129 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:33.848338 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781957973848326 us; error 25 us; skew 500 ppm
Jun 20 12:19:33 dist-test-slave-9b3m krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:33.851146 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:33.852103 1475 webserver.cc:460] Webserver started at http://127.0.98.62:42749/ using document root <none> and password file <none>
I20260620 12:19:33.852708 1475 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:33.852800 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:33.853029 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:33.855103 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "032d23a61d5b45afa0bdde126cefd2ae"
format_stamp: "Formatted at 2026-06-20 12:19:33 on dist-test-slave-9b3m"
server_key: "cbc5f23a9b1a28b33942f9434677ede1"
server_key_iv: "909e7366221c676fe52a6c0cedfcc913"
server_key_version: "encryptionkey@0"
I20260620 12:19:33.855759 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "032d23a61d5b45afa0bdde126cefd2ae"
format_stamp: "Formatted at 2026-06-20 12:19:33 on dist-test-slave-9b3m"
server_key: "cbc5f23a9b1a28b33942f9434677ede1"
server_key_iv: "909e7366221c676fe52a6c0cedfcc913"
server_key_version: "encryptionkey@0"
I20260620 12:19:33.859411 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 12:19:33.861851 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:33.863045 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:19:33.863193 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "032d23a61d5b45afa0bdde126cefd2ae"
format_stamp: "Formatted at 2026-06-20 12:19:33 on dist-test-slave-9b3m"
server_key: "cbc5f23a9b1a28b33942f9434677ede1"
server_key_iv: "909e7366221c676fe52a6c0cedfcc913"
server_key_version: "encryptionkey@0"
I20260620 12:19:33.863332 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:33.889917 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:33.893168 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:33.893383 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:33.901139 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34727
I20260620 12:19:33.901145 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34727 every 8 connection(s)
I20260620 12:19:33.902057 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:33.902469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 1475
I20260620 12:19:33.902586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:33.902868 392 external_mini_cluster.cc:1428] Setting key e1efd810b13002991368d3696c5dc7cb
I20260620 12:19:33.905258 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:33.911046 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae: Bootstrap starting.
Jun 20 12:19:33 dist-test-slave-9b3m krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:33.913934 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:33.915216 1553 log.cc:824] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:33.917317 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae: No bootstrap required, opened a new log
I20260620 12:19:33.917382 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:33.904333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.0.98.62:34727)
0620 12:19:33.904777 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:33.904791 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:33.904825 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:19:33.905573 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:33.905591 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:33.905921 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:33.906253 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:33.907009 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:33.907996 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:33.908858 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:33.909415 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:33.909587 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:33.912607 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:33.912612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:33.912625 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:33.912653 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:33.914495 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:33.914998 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:33.915002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:33.915006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:33.915110 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:33.915439 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:33.915442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:33.915444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:33.915821 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:33.916010 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:33.916340 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 12:19:33.916611 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":140,"threads_started":1}
I20260620 12:19:33.920691 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d23a61d5b45afa0bdde126cefd2ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260620 12:19:33.920972 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:33.921087 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032d23a61d5b45afa0bdde126cefd2ae, State: Initialized, Role: FOLLOWER
I20260620 12:19:33.921550 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d23a61d5b45afa0bdde126cefd2ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260620 12:19:33.921702 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:33.921792 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:33.921893 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:33.922796 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d23a61d5b45afa0bdde126cefd2ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260620 12:19:33.923131 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032d23a61d5b45afa0bdde126cefd2ae; no voters:
I20260620 12:19:33.923437 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:33.923580 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:33.923823 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [term 1 LEADER]: Becoming Leader. State: Replica: 032d23a61d5b45afa0bdde126cefd2ae, State: Running, Role: LEADER
I20260620 12:19:33.924154 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d23a61d5b45afa0bdde126cefd2ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260620 12:19:33.924640 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:33.925837 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032d23a61d5b45afa0bdde126cefd2ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d23a61d5b45afa0bdde126cefd2ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } } }
I20260620 12:19:33.925855 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032d23a61d5b45afa0bdde126cefd2ae. Latest consensus state: current_term: 1 leader_uuid: "032d23a61d5b45afa0bdde126cefd2ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d23a61d5b45afa0bdde126cefd2ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } } }
I20260620 12:19:33.926057 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:33.926090 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:33.926427 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:33.929448 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:33.934804 1567 catalog_manager.cc:1347] Generated new cluster ID: 09e2c376db5340b49725d8a0f92326c6
I20260620 12:19:33.934862 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:33.947594 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:33.948628 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:33.955147 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae: Generated new TSK 0
I20260620 12:19:33.955809 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:33.983922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:34.095146 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:34.095382 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:34.095424 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:34.099476 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:34.099525 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:34.099562 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:19:34.099637 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:34.104624 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:34.105650 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:34.106612 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:34.116241 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:34.116241 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:34.116259 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:34.116580 1579 server_base.cc:1034] running on GCE node
I20260620 12:19:34.116950 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:34.117552 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:34.118723 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781957974118707 us; error 27 us; skew 500 ppm
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:34.121450 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:34.122256 1579 webserver.cc:460] Webserver started at http://127.0.98.1:32915/ using document root <none> and password file <none>
I20260620 12:19:34.122787 1579 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:34.122875 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:34.123096 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:34.125169 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "0675d73fdbeb494d83896c434f103587"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "1933936db06c09113c1fa55823630e62"
server_key_iv: "c34d7fbb092b418f67de0dd4a3406ce2"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.125810 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0675d73fdbeb494d83896c434f103587"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "1933936db06c09113c1fa55823630e62"
server_key_iv: "c34d7fbb092b418f67de0dd4a3406ce2"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.129568 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:34.131880 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:34.133111 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:34.133250 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "0675d73fdbeb494d83896c434f103587"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "1933936db06c09113c1fa55823630e62"
server_key_iv: "c34d7fbb092b418f67de0dd4a3406ce2"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.133358 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:34.151525 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:34.154662 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:34.154873 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:34.156399 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:34.156476 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:34.156539 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:34.156569 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:34.167263 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39319
I20260620 12:19:34.167604 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39319 every 8 connection(s)
I20260620 12:19:34.168335 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:34.169554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 1579
I20260620 12:19:34.169670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:34.169950 392 external_mini_cluster.cc:1428] Setting key 3319b9479a46233b16358f7209492448
I20260620 12:19:34.171327 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.157710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.1:57587)
0620 12:19:34.158173 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.158227 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.159228 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.159842 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.159849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.160249 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:34.160865 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.160889 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.161804 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.161808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.162759 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.162768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.163441 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.163445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.163630 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.164227 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:34.164244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:34.166002 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:34.168244 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.168249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.168261 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.168588 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.168951 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.168955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.168960 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.169154 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.169573 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:34.169581 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:34.169873 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.170062 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.170381 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":139,"threads_started":1}
I20260620 12:19:34.172294 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.157268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57587 (local address 127.0.98.62:34727)
0620 12:19:34.157443 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.157450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.158348 (+ 898us) server_negotiation.cc:407] Connection header received
0620 12:19:34.159493 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.159496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.159557 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.159666 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:34.161033 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.161667 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.162892 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.163265 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.163389 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.166180 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:34.166184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:34.166189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:34.166220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:34.168127 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.168727 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.168731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.168736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.168828 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.169260 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.169269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.169270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.169451 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:34.169544 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.171922 (+ 2378us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.172048 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:19:34.173594 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34727
I20260620 12:19:34.173854 1710 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:34.174398 1710 heartbeater.cc:511] Master 127.0.98.62:34727 requested a full tablet report, sending...
I20260620 12:19:34.176088 1506 ts_manager.cc:194] Registered new tserver with Master: 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319)
I20260620 12:19:34.177956 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57587
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:34.186204 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.177355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34121 (local address 127.0.98.62:34727)
0620 12:19:34.177458 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.177462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.178880 (+ 1418us) server_negotiation.cc:407] Connection header received
0620 12:19:34.179010 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.179013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.179057 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.179133 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:34.179712 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.180400 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.181578 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.181782 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.181908 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.182887 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:34.182889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:34.182891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:34.182910 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:34.184346 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.184921 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.184926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.184929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.185025 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.185411 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.185417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.185419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.185617 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:34.185736 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.186019 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.186104 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:19:34.186471 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.177315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.1:34121)
0620 12:19:34.178775 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.178794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.178905 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.179184 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.179189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.179425 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:34.179582 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.179590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.180540 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.180544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.181464 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.181474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.181920 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.181929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.181999 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.182433 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:34.182444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:34.182760 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:34.184461 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.184468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.184474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.184771 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.185229 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.185231 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.185232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.185309 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.185725 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:34.185728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:34.185840 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.185956 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.186147 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1358,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:34.199440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:34.305501 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:34.305819 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:34.305898 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:34.309788 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:34.309864 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:34.309933 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:19:34.310010 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:34.315330 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:34.316272 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:34.317296 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:34.326920 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:34.326865 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:34.326865 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:34.326880 1719 server_base.cc:1034] running on GCE node
I20260620 12:19:34.327467 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:34.328029 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:34.329221 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781957974329202 us; error 35 us; skew 500 ppm
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:34.331843 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:34.332707 1719 webserver.cc:460] Webserver started at http://127.0.98.2:42287/ using document root <none> and password file <none>
I20260620 12:19:34.333315 1719 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:34.333400 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:34.333619 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:34.335566 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "32ebcb049b1146f29203690b26c74fa6"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "9c62f59cda95a113a04bf719e8423720"
server_key_iv: "c4a0aede0186124df0e342eeae3833c7"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.336110 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32ebcb049b1146f29203690b26c74fa6"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "9c62f59cda95a113a04bf719e8423720"
server_key_iv: "c4a0aede0186124df0e342eeae3833c7"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.339560 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:19:34.341866 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:34.342941 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:19:34.343070 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "32ebcb049b1146f29203690b26c74fa6"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "9c62f59cda95a113a04bf719e8423720"
server_key_iv: "c4a0aede0186124df0e342eeae3833c7"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.343200 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:34.352615 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:34.355443 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:34.355633 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:34.357154 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:34.357219 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:34.357281 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:34.357311 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:34.367942 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35747
I20260620 12:19:34.368168 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35747 every 8 connection(s)
I20260620 12:19:34.368805 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:34.370532 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.357754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40659 (local address 127.0.98.62:34727)
0620 12:19:34.357894 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.357899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.358832 (+ 933us) server_negotiation.cc:407] Connection header received
0620 12:19:34.360117 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.360123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.360190 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.360254 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:34.362043 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.362648 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.363771 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.363997 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.364112 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.366957 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:34.366961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:34.366963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:34.366990 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:34.368574 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.369143 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.369148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.369150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.369237 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.369606 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.369609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.369611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.369795 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:34.369876 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.370326 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.370418 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:19:34.371243 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.358178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.2:40659)
0620 12:19:34.358651 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.358704 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.359829 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.360694 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.360706 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.361187 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:34.361896 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.361919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.362783 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.362786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.363629 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.363638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.364158 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.364162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.364325 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.364850 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:34.364866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:34.366781 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:34.368690 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.368703 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.368723 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.369020 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.369330 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.369335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.369337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.369489 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.369904 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:34.369908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:34.370110 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.370252 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.370468 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260620 12:19:34.372412 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34727
I20260620 12:19:34.372565 1850 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:34.373159 1850 heartbeater.cc:511] Master 127.0.98.62:34727 requested a full tablet report, sending...
I20260620 12:19:34.374193 1507 ts_manager.cc:194] Registered new tserver with Master: 32ebcb049b1146f29203690b26c74fa6 (127.0.98.2:35747)
I20260620 12:19:34.374979 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40659
I20260620 12:19:34.375041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 1719
I20260620 12:19:34.375161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:34.375456 392 external_mini_cluster.cc:1428] Setting key b648dfb6f0bf8b398a61dd33c2681d0a
I20260620 12:19:34.382942 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.374380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39307 (local address 127.0.98.62:34727)
0620 12:19:34.374513 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.374517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.374734 (+ 217us) server_negotiation.cc:407] Connection header received
0620 12:19:34.374852 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.374858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.374909 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.374989 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:34.377032 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.377553 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.378356 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.378586 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.378695 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.379715 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:34.379719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:34.379721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:34.379744 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:34.381338 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.381761 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.381764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.381766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.381846 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.382121 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.382124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.382126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.382321 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:34.382430 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.382687 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.382789 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:19:34.382951 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.374348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.2:39307)
0620 12:19:34.374650 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.374664 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.374753 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.375024 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.375027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.375290 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:34.375412 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.375418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.377686 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.377689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.378239 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.378247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.378695 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.378698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.378786 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.379265 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:34.379276 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:34.379566 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:34.381461 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.381464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.381466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.381662 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.381937 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.381941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.381942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.382025 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.382419 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:34.382422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:34.382508 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.382636 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.382776 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:19:34.400111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:34.514405 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:34.514688 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:34.514762 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:34.518843 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:34.518920 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:34.518982 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:19:34.519054 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:34.525923 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:34.526993 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:34.528025 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:34.537731 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:34.537936 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:34.537976 1859 server_base.cc:1034] running on GCE node
W20260620 12:19:34.538071 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:34.538501 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:34.539173 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:34.540364 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781957974540345 us; error 37 us; skew 500 ppm
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:34.543299 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:34.544185 1859 webserver.cc:460] Webserver started at http://127.0.98.3:36983/ using document root <none> and password file <none>
I20260620 12:19:34.544761 1859 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:34.544852 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:34.545132 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:34.547233 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "d9543f151f1e416f959ac206a5e6adcf"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "940e819f5902e69a73283da26217e12c"
server_key_iv: "02a8d5d0aa792dfe14ffecededfb73c0"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.547782 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d9543f151f1e416f959ac206a5e6adcf"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "940e819f5902e69a73283da26217e12c"
server_key_iv: "02a8d5d0aa792dfe14ffecededfb73c0"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.551369 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:34.553809 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:34.555045 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:34.555178 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "d9543f151f1e416f959ac206a5e6adcf"
format_stamp: "Formatted at 2026-06-20 12:19:34 on dist-test-slave-9b3m"
server_key: "940e819f5902e69a73283da26217e12c"
server_key_iv: "02a8d5d0aa792dfe14ffecededfb73c0"
server_key_version: "encryptionkey@0"
I20260620 12:19:34.555279 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:34.576902 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:34.579869 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:34.580065 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:34.581588 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:34.581667 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:34.581737 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:34.581784 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:34.592559 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40811
I20260620 12:19:34.592904 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40811 every 8 connection(s)
I20260620 12:19:34.593678 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:34.595618 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.582406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53657 (local address 127.0.98.62:34727)
0620 12:19:34.582574 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.582581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.583421 (+ 840us) server_negotiation.cc:407] Connection header received
0620 12:19:34.584485 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.584490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.584556 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.584623 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:34.586258 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.586822 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.588090 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.588323 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.588464 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.591599 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:34.591603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:34.591606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:34.591632 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:34.593441 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.594055 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.594061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.594064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.594168 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.594554 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.594558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.594559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.594767 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:34.594851 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.595385 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.595486 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 12:19:34.596189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 1859
I20260620 12:19:34.596314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:34.596423 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.582786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.3:53657)
0620 12:19:34.583240 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.583291 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.584222 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.585007 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.585014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.585452 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:34.586036 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.586059 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.586929 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.586933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.587952 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.587962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.588509 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.588517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.588764 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.589480 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:34.589499 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:34.591444 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:34.593572 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.593577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.593588 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.593946 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.594284 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.594288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.594291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.594451 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.594864 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:34.594868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:34.595088 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.595294 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.595528 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":143,"threads_started":1}
I20260620 12:19:34.596619 392 external_mini_cluster.cc:1428] Setting key be24abb57328ccb059021788483dcb06
I20260620 12:19:34.597815 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34727
I20260620 12:19:34.597934 1990 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:34.598451 1990 heartbeater.cc:511] Master 127.0.98.62:34727 requested a full tablet report, sending...
I20260620 12:19:34.599632 1506 ts_manager.cc:194] Registered new tserver with Master: d9543f151f1e416f959ac206a5e6adcf (127.0.98.3:40811)
I20260620 12:19:34.600349 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53657
I20260620 12:19:34.606837 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.599949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42001 (local address 127.0.98.62:34727)
0620 12:19:34.600070 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.600074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.600277 (+ 203us) server_negotiation.cc:407] Connection header received
0620 12:19:34.600401 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.600403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.600447 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.600537 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:34.600933 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.601702 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.602528 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.602727 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.602808 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.603832 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:34.603835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:34.603837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:34.603860 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:34.605494 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.605904 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.605907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.605908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.605983 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.606251 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.606253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.606255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.606403 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:34.606477 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.606655 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.606734 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:19:34.606854 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.599886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.3:42001)
0620 12:19:34.600190 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.600202 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.600314 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.600535 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.600538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.600735 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:34.600821 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.600826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.601839 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.601842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.602411 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.602419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.602824 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.602826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.602894 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.603350 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:34.603361 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:34.603694 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:34.605609 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.605612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.605613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.605798 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.606075 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.606077 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.606078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.606148 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.606478 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:34.606480 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:34.606549 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.606623 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.606721 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260620 12:19:34.610148 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:34.644766 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.635996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.0.98.62:34727)
0620 12:19:34.636168 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.636173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.636190 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:19:34.636252 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.636255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.636312 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.636438 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:34.637520 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.638149 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.639021 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.639260 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.639373 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.641924 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:34.641927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:34.641929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:34.641953 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:34.643383 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.643760 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.643763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.643764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.643839 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.644063 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.644065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.644066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.644253 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:34.644353 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.644556 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.644646 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:19:34.647769 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:19:34.650910 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:34.664965 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.659374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.0.98.2:35747)
0620 12:19:34.659943 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.659955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.659978 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:19:34.660061 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.660070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.660292 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.660468 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:34.660783 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.661955 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.663535 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.664272 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.664395 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.664560 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.664724 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.664816 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":119,"threads_started":1}
I20260620 12:19:34.664955 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.659211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35747 (local address 127.0.0.1:57320)
0620 12:19:34.659700 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.659746 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.659871 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.660474 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.660480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.660505 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:34.660643 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.660652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.662066 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.662069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.663393 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.663403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.664396 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.664399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.664499 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.664533 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.664691 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.664811 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":25,"thread_start_us":72,"threads_started":1}
I20260620 12:19:34.665356 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.659964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.0.98.3:40811)
0620 12:19:34.660271 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.660285 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.660562 (+ 277us) server_negotiation.cc:407] Connection header received
0620 12:19:34.660650 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.660654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.660861 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.660996 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:34.661375 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.662324 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.663593 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.664330 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.664428 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.664571 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.665025 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.665220 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":116,"threads_started":1}
I20260620 12:19:34.665620 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.659368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40811 (local address 127.0.0.1:43134)
0620 12:19:34.660375 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.660393 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.660475 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.661169 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.661172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.661186 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:34.661261 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.661265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.662447 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.662451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.663465 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.663474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.664633 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.664637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.664729 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.664756 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.664834 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.664902 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":8,"thread_start_us":93,"threads_started":1}
I20260620 12:19:34.665761 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.660221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.0.98.1:39319)
0620 12:19:34.661142 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.661155 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.661177 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:19:34.661246 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.661251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.661405 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.661544 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:34.661750 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.662627 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.664027 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.664770 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.664859 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.664991 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.665524 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.665607 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":58,"threads_started":1}
I20260620 12:19:34.666057 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.659882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39319 (local address 127.0.0.1:36618)
0620 12:19:34.660806 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.660821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.660899 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.661535 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.661538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.661556 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:34.661638 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.661644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.662753 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.662756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.663894 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.663903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.665213 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.665216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.665304 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.665323 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.665399 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.665960 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260620 12:19:34.668378 1924 tablet_service.cc:1467] Processing CreateTablet for tablet b50858c5f48646cc9b2c0a0d94aa7381 (DEFAULT_TABLE table=test-table [id=d5bbea2657fd45439ca80ae86f2b4482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:34.668377 1784 tablet_service.cc:1467] Processing CreateTablet for tablet b50858c5f48646cc9b2c0a0d94aa7381 (DEFAULT_TABLE table=test-table [id=d5bbea2657fd45439ca80ae86f2b4482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:34.669356 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50858c5f48646cc9b2c0a0d94aa7381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.669365 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50858c5f48646cc9b2c0a0d94aa7381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.676110 2013 tablet_bootstrap.cc:492] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: Bootstrap starting.
I20260620 12:19:34.677284 2014 tablet_bootstrap.cc:492] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: Bootstrap starting.
I20260620 12:19:34.678426 2013 tablet_bootstrap.cc:654] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.678364 1644 tablet_service.cc:1467] Processing CreateTablet for tablet b50858c5f48646cc9b2c0a0d94aa7381 (DEFAULT_TABLE table=test-table [id=d5bbea2657fd45439ca80ae86f2b4482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:34.679270 2014 tablet_bootstrap.cc:654] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.679283 2013 log.cc:824] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:34.679358 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50858c5f48646cc9b2c0a0d94aa7381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.680001 2014 log.cc:824] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:34.681408 2013 tablet_bootstrap.cc:492] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: No bootstrap required, opened a new log
I20260620 12:19:34.681604 2013 ts_tablet_manager.cc:1397] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 12:19:34.681988 2014 tablet_bootstrap.cc:492] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: No bootstrap required, opened a new log
I20260620 12:19:34.682165 2014 ts_tablet_manager.cc:1397] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:19:34.684480 2013 raft_consensus.cc:348] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.684719 2013 raft_consensus.cc:374] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.684778 2013 raft_consensus.cc:729] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9543f151f1e416f959ac206a5e6adcf, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.685230 2013 consensus_queue.cc:260] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.685850 1990 heartbeater.cc:503] Master 127.0.98.62:34727 was elected leader, sending a full tablet report...
I20260620 12:19:34.685977 2014 raft_consensus.cc:348] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.686224 2014 raft_consensus.cc:374] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.686282 2014 raft_consensus.cc:729] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ebcb049b1146f29203690b26c74fa6, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.686549 2018 tablet_bootstrap.cc:492] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: Bootstrap starting.
I20260620 12:19:34.686658 2014 consensus_queue.cc:260] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.687614 2014 ts_tablet_manager.cc:1428] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:19:34.688884 1850 heartbeater.cc:503] Master 127.0.98.62:34727 was elected leader, sending a full tablet report...
I20260620 12:19:34.689121 2018 tablet_bootstrap.cc:654] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.690295 2013 ts_tablet_manager.cc:1428] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20260620 12:19:34.690421 2018 log.cc:824] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:34.692359 2018 tablet_bootstrap.cc:492] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: No bootstrap required, opened a new log
I20260620 12:19:34.692540 2018 ts_tablet_manager.cc:1397] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:19:34.695137 2018 raft_consensus.cc:348] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.695335 2018 raft_consensus.cc:374] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.695372 2018 raft_consensus.cc:729] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0675d73fdbeb494d83896c434f103587, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.695716 2018 consensus_queue.cc:260] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.696208 1710 heartbeater.cc:503] Master 127.0.98.62:34727 was elected leader, sending a full tablet report...
I20260620 12:19:34.696540 2018 ts_tablet_manager.cc:1428] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 12:19:34.737322 2019 raft_consensus.cc:482] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:34.737572 2019 raft_consensus.cc:504] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.738731 2019 leader_election.cc:288] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9543f151f1e416f959ac206a5e6adcf (127.0.98.3:40811), 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319)
I20260620 12:19:34.743221 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.738960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39319 (local address 127.0.98.2:55737)
0620 12:19:34.739096 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.739113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.739223 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.739537 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.739539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.739557 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:34.739703 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.739710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.741198 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.741201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.742228 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.742237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.742901 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.742903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.742978 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.742998 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.743055 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.743111 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 12:19:34.743287 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.739056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55737 (local address 127.0.98.1:39319)
0620 12:19:34.739177 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.739183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.739197 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:19:34.739343 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.739347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.739421 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.739500 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:34.739817 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.741093 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.742359 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.742791 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.742897 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.742962 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.743092 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.743165 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:19:34.743922 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50858c5f48646cc9b2c0a0d94aa7381" candidate_uuid: "32ebcb049b1146f29203690b26c74fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0675d73fdbeb494d83896c434f103587" is_pre_election: true
I20260620 12:19:34.743986 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.739154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55779 (local address 127.0.98.3:40811)
0620 12:19:34.739288 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.739293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.739658 (+ 365us) server_negotiation.cc:407] Connection header received
0620 12:19:34.739714 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.739717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.739788 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.739865 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:34.741085 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.741608 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.742788 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.743402 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.743519 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.743572 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.743819 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.743884 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:19:34.744014 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.738960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40811 (local address 127.0.98.2:55779)
0620 12:19:34.739542 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.739555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.739641 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.739885 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.739888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.739904 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:34.740927 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.740936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.741726 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.741730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.742676 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.742682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.743524 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.743528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.743639 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.743656 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.743800 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.743870 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":259,"thread_start_us":135,"threads_started":1}
I20260620 12:19:34.744269 1664 raft_consensus.cc:2454] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ebcb049b1146f29203690b26c74fa6 in term 0.
I20260620 12:19:34.744566 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50858c5f48646cc9b2c0a0d94aa7381" candidate_uuid: "32ebcb049b1146f29203690b26c74fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9543f151f1e416f959ac206a5e6adcf" is_pre_election: true
I20260620 12:19:34.744756 1736 leader_election.cc:302] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0675d73fdbeb494d83896c434f103587, 32ebcb049b1146f29203690b26c74fa6; no voters:
I20260620 12:19:34.744854 1944 raft_consensus.cc:2454] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ebcb049b1146f29203690b26c74fa6 in term 0.
I20260620 12:19:34.745024 2019 raft_consensus.cc:2781] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:34.745155 2019 raft_consensus.cc:482] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:34.745221 2019 raft_consensus.cc:3037] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:34.746330 2019 raft_consensus.cc:504] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.746739 2019 leader_election.cc:288] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [CANDIDATE]: Term 1 election: Requested vote from peers d9543f151f1e416f959ac206a5e6adcf (127.0.98.3:40811), 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319)
I20260620 12:19:34.747026 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50858c5f48646cc9b2c0a0d94aa7381" candidate_uuid: "32ebcb049b1146f29203690b26c74fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9543f151f1e416f959ac206a5e6adcf"
I20260620 12:19:34.747098 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50858c5f48646cc9b2c0a0d94aa7381" candidate_uuid: "32ebcb049b1146f29203690b26c74fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0675d73fdbeb494d83896c434f103587"
I20260620 12:19:34.747136 1944 raft_consensus.cc:3037] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:34.747243 1664 raft_consensus.cc:3037] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:34.748188 1944 raft_consensus.cc:2454] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ebcb049b1146f29203690b26c74fa6 in term 1.
I20260620 12:19:34.748414 1664 raft_consensus.cc:2454] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ebcb049b1146f29203690b26c74fa6 in term 1.
I20260620 12:19:34.748476 1738 leader_election.cc:302] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ebcb049b1146f29203690b26c74fa6, d9543f151f1e416f959ac206a5e6adcf; no voters:
I20260620 12:19:34.748661 2019 raft_consensus.cc:2781] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:34.748894 2019 raft_consensus.cc:686] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 32ebcb049b1146f29203690b26c74fa6, State: Running, Role: LEADER
I20260620 12:19:34.749253 2019 consensus_queue.cc:237] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } }
I20260620 12:19:34.752357 1507 catalog_manager.cc:5495] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32ebcb049b1146f29203690b26c74fa6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } health_report { overall_health: UNKNOWN } } }
I20260620 12:19:34.767102 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.763585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.98.2:35747)
0620 12:19:34.763736 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.763739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.763752 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:19:34.763930 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.763932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.763981 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.764075 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:34.764314 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.764799 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.765684 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.765897 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.765975 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.766315 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:34.766682 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:34.766812 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.766914 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.766964 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:19:34.772526 1664 raft_consensus.cc:1261] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Refusing update from remote peer 32ebcb049b1146f29203690b26c74fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:34.772764 1944 raft_consensus.cc:1261] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Refusing update from remote peer 32ebcb049b1146f29203690b26c74fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:34.773299 2023 consensus_queue.cc:1035] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:34.773478 2019 consensus_queue.cc:1035] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:34.782496 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298899864661458944 to 7298899864575913984. Current Snapshot: MvccSnapshot[applied={T|T < 7298899864661458944}]
I20260620 12:19:34.783406 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298899864661458944 to 7298899864575913984. Current Snapshot: MvccSnapshot[applied={T|T < 7298899864661458944}]
I20260620 12:19:34.792142 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:34.792328 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:34.794620 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae: Sending DeleteTablet for 3 replicas of tablet b50858c5f48646cc9b2c0a0d94aa7381
I20260620 12:19:34.795464 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet b50858c5f48646cc9b2c0a0d94aa7381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43134
I20260620 12:19:34.795460 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet b50858c5f48646cc9b2c0a0d94aa7381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57320
I20260620 12:19:34.796062 2035 tablet_replica.cc:330] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: stopping tablet replica
I20260620 12:19:34.796082 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet b50858c5f48646cc9b2c0a0d94aa7381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36618
I20260620 12:19:34.796103 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:19:34.796516 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:34.796530 2037 tablet_replica.cc:330] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: stopping tablet replica
I20260620 12:19:34.796743 2035 raft_consensus.cc:2229] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:19:34.796765 2037 raft_consensus.cc:2229] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:34.797144 2035 raft_consensus.cc:2258] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:34.797223 2036 tablet_replica.cc:330] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: stopping tablet replica
I20260620 12:19:34.797494 2036 raft_consensus.cc:2229] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:34.797778 2036 raft_consensus.cc:2258] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:34.797955 2037 raft_consensus.cc:2258] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:34.799094 2037 ts_tablet_manager.cc:1905] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:34.799094 2035 ts_tablet_manager.cc:1905] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:34.801545 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 17ef64ea250b4527984f489d697fb725 (DEFAULT_TABLE table=test-table [id=cecd1942143e4898b3f9773a92da0e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:34.801649 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 17ef64ea250b4527984f489d697fb725 (DEFAULT_TABLE table=test-table [id=cecd1942143e4898b3f9773a92da0e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:34.801798 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ef64ea250b4527984f489d697fb725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.801892 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ef64ea250b4527984f489d697fb725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.801937 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 17ef64ea250b4527984f489d697fb725 (DEFAULT_TABLE table=test-table [id=cecd1942143e4898b3f9773a92da0e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:34.802171 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ef64ea250b4527984f489d697fb725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.803951 2018 tablet_bootstrap.cc:492] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: Bootstrap starting.
I20260620 12:19:34.803978 2014 tablet_bootstrap.cc:492] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: Bootstrap starting.
I20260620 12:19:34.804404 2013 tablet_bootstrap.cc:492] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: Bootstrap starting.
I20260620 12:19:34.804872 2018 tablet_bootstrap.cc:654] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.804872 2014 tablet_bootstrap.cc:654] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.805302 2013 tablet_bootstrap.cc:654] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.806352 2035 ts_tablet_manager.cc:1918] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:19:34.806491 2035 log.cc:1196] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/b50858c5f48646cc9b2c0a0d94aa7381
I20260620 12:19:34.806883 2035 ts_tablet_manager.cc:1939] T b50858c5f48646cc9b2c0a0d94aa7381 P 32ebcb049b1146f29203690b26c74fa6: Deleting consensus metadata
I20260620 12:19:34.807003 2036 ts_tablet_manager.cc:1905] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:34.807705 2013 tablet_bootstrap.cc:492] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: No bootstrap required, opened a new log
I20260620 12:19:34.807789 2013 ts_tablet_manager.cc:1397] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 12:19:34.808208 2013 raft_consensus.cc:348] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.808235 1492 catalog_manager.cc:4841] TS 32ebcb049b1146f29203690b26c74fa6 (127.0.98.2:35747): tablet b50858c5f48646cc9b2c0a0d94aa7381 (table test-table [id=d5bbea2657fd45439ca80ae86f2b4482]) successfully deleted
I20260620 12:19:34.808318 2013 raft_consensus.cc:374] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.808369 2013 raft_consensus.cc:729] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9543f151f1e416f959ac206a5e6adcf, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.808488 2013 consensus_queue.cc:260] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.808748 2013 ts_tablet_manager.cc:1428] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:34.809211 2014 tablet_bootstrap.cc:492] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: No bootstrap required, opened a new log
I20260620 12:19:34.809290 2014 ts_tablet_manager.cc:1397] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 12:19:34.809783 2014 raft_consensus.cc:348] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.809880 2014 raft_consensus.cc:374] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.809907 2014 raft_consensus.cc:729] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ebcb049b1146f29203690b26c74fa6, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.810012 2014 consensus_queue.cc:260] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.810256 2014 ts_tablet_manager.cc:1428] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:19:34.810518 2037 ts_tablet_manager.cc:1918] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:19:34.810603 2037 log.cc:1196] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/b50858c5f48646cc9b2c0a0d94aa7381
I20260620 12:19:34.811014 2036 ts_tablet_manager.cc:1918] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:19:34.811143 2036 log.cc:1196] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/b50858c5f48646cc9b2c0a0d94aa7381
I20260620 12:19:34.811491 2036 ts_tablet_manager.cc:1939] T b50858c5f48646cc9b2c0a0d94aa7381 P d9543f151f1e416f959ac206a5e6adcf: Deleting consensus metadata
I20260620 12:19:34.811532 2018 tablet_bootstrap.cc:492] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: No bootstrap required, opened a new log
I20260620 12:19:34.811609 2018 ts_tablet_manager.cc:1397] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260620 12:19:34.812045 2018 raft_consensus.cc:348] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.812166 2018 raft_consensus.cc:374] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.812212 2018 raft_consensus.cc:729] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0675d73fdbeb494d83896c434f103587, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.812338 2018 consensus_queue.cc:260] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.812587 2018 ts_tablet_manager.cc:1428] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:19:34.812979 1494 catalog_manager.cc:4841] TS d9543f151f1e416f959ac206a5e6adcf (127.0.98.3:40811): tablet b50858c5f48646cc9b2c0a0d94aa7381 (table test-table [id=d5bbea2657fd45439ca80ae86f2b4482]) successfully deleted
I20260620 12:19:34.813697 2037 ts_tablet_manager.cc:1939] T b50858c5f48646cc9b2c0a0d94aa7381 P 0675d73fdbeb494d83896c434f103587: Deleting consensus metadata
I20260620 12:19:34.814755 1492 catalog_manager.cc:4841] TS 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319): tablet b50858c5f48646cc9b2c0a0d94aa7381 (table test-table [id=d5bbea2657fd45439ca80ae86f2b4482]) successfully deleted
W20260620 12:19:34.844789 1991 tablet.cc:2369] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:19:34.870285 1851 tablet.cc:2369] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:19:34.897253 2017 raft_consensus.cc:482] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:34.897413 2017 raft_consensus.cc:504] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.898618 2017 leader_election.cc:288] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319), 32ebcb049b1146f29203690b26c74fa6 (127.0.98.2:35747)
I20260620 12:19:34.903474 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.898909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39319 (local address 127.0.98.3:44711)
0620 12:19:34.899017 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.899033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.899137 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.899516 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.899521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.899547 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:34.899687 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.899695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.900671 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.900675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.902044 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.902056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.903037 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.903040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.903155 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.903178 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.903252 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.903322 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":98,"spinlock_wait_cycles":20480}
I20260620 12:19:34.903569 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.899632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48053 (local address 127.0.98.2:35747)
0620 12:19:34.899770 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.899776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.899790 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:19:34.899837 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.899839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.899881 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.899958 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:34.900239 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.900907 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.902348 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.902911 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.902983 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.903043 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.903377 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.903458 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:19:34.903955 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.899229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35747 (local address 127.0.98.3:48053)
0620 12:19:34.899488 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.899502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.899608 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.899985 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.899988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.900007 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:34.900114 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.900129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.901030 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.901034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.902215 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.902225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.903135 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.903138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.903244 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.903264 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.903745 (+ 481us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.903814 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":98,"threads_started":1}
I20260620 12:19:34.903951 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.899000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44711 (local address 127.0.98.1:39319)
0620 12:19:34.899176 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.899182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.899200 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:19:34.899256 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.899259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.899307 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.899397 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:34.899822 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.900559 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.902216 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.902796 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.902905 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.902977 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.903761 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.903840 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 12:19:34.904388 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ef64ea250b4527984f489d697fb725" candidate_uuid: "d9543f151f1e416f959ac206a5e6adcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0675d73fdbeb494d83896c434f103587" is_pre_election: true
I20260620 12:19:34.904532 1664 raft_consensus.cc:2454] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9543f151f1e416f959ac206a5e6adcf in term 0.
I20260620 12:19:34.904692 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ef64ea250b4527984f489d697fb725" candidate_uuid: "d9543f151f1e416f959ac206a5e6adcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ebcb049b1146f29203690b26c74fa6" is_pre_election: true
I20260620 12:19:34.904911 1804 raft_consensus.cc:2454] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9543f151f1e416f959ac206a5e6adcf in term 0.
I20260620 12:19:34.904919 1876 leader_election.cc:302] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0675d73fdbeb494d83896c434f103587, d9543f151f1e416f959ac206a5e6adcf; no voters:
I20260620 12:19:34.905508 2017 raft_consensus.cc:2781] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:34.905583 2017 raft_consensus.cc:482] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:34.905625 2017 raft_consensus.cc:3037] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:34.906895 2017 raft_consensus.cc:504] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.907282 2017 leader_election.cc:288] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [CANDIDATE]: Term 1 election: Requested vote from peers 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319), 32ebcb049b1146f29203690b26c74fa6 (127.0.98.2:35747)
I20260620 12:19:34.907729 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ef64ea250b4527984f489d697fb725" candidate_uuid: "d9543f151f1e416f959ac206a5e6adcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0675d73fdbeb494d83896c434f103587"
I20260620 12:19:34.907739 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ef64ea250b4527984f489d697fb725" candidate_uuid: "d9543f151f1e416f959ac206a5e6adcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ebcb049b1146f29203690b26c74fa6"
I20260620 12:19:34.907845 1804 raft_consensus.cc:3037] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:34.907848 1664 raft_consensus.cc:3037] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:34.909020 1804 raft_consensus.cc:2454] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9543f151f1e416f959ac206a5e6adcf in term 1.
I20260620 12:19:34.909026 1664 raft_consensus.cc:2454] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9543f151f1e416f959ac206a5e6adcf in term 1.
I20260620 12:19:34.909475 1876 leader_election.cc:302] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0675d73fdbeb494d83896c434f103587, d9543f151f1e416f959ac206a5e6adcf; no voters:
I20260620 12:19:34.909749 2017 raft_consensus.cc:2781] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:34.909960 2017 raft_consensus.cc:686] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 1 LEADER]: Becoming Leader. State: Replica: d9543f151f1e416f959ac206a5e6adcf, State: Running, Role: LEADER
I20260620 12:19:34.910458 2017 consensus_queue.cc:237] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.912883 1507 catalog_manager.cc:5495] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf reported cstate change: term changed from 0 to 1, leader changed from <none> to d9543f151f1e416f959ac206a5e6adcf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d9543f151f1e416f959ac206a5e6adcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } health_report { overall_health: HEALTHY } } }
W20260620 12:19:34.919760 1711 tablet.cc:2369] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 12:19:34 dist-test-slave-9b3m krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:34.967031 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.958948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.0.1:48716)
0620 12:19:34.959282 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:34.959296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:34.959434 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:34.959720 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:34.959722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:34.960087 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:34.960183 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.960189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.960884 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.960886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.961446 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:34.961452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.961892 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.961894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:34.961951 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.962453 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:34.962464 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:34.963799 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:34.965559 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.965566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.965570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.965792 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.966105 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:34.966110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:34.966112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:34.966199 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:34.966561 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:34.966566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:34.966673 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 12:19:34.966761 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:19:34.966861 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":128,"threads_started":1}
I20260620 12:19:34.967029 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:34.958983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48716 (local address 127.0.98.62:34727)
0620 12:19:34.959129 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:34.959134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:34.959378 (+ 244us) server_negotiation.cc:407] Connection header received
0620 12:19:34.959525 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:34.959527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:34.959567 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:34.959635 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:34.960287 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.960798 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.961585 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:34.961791 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:34.961880 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:34.963946 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:34.963949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:34.963950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:34.963973 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:34.965468 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.965906 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.965909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.965910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.965983 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:34.966282 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:34.966284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:34.966286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:34.966445 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:34.966575 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:34.966820 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 12:19:34.966909 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:19:34.970669 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48716:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:19:34.971287 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:34.975997 1644 tablet_service.cc:1467] Processing CreateTablet for tablet f696e1a2fda44d9fae364f65d666fd76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a85f01a53e964ccd87295fd3e83cd677]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:19:34.976032 1924 tablet_service.cc:1467] Processing CreateTablet for tablet f696e1a2fda44d9fae364f65d666fd76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a85f01a53e964ccd87295fd3e83cd677]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:19:34.976421 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f696e1a2fda44d9fae364f65d666fd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.976439 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f696e1a2fda44d9fae364f65d666fd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.978892 2013 tablet_bootstrap.cc:492] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: Bootstrap starting.
I20260620 12:19:34.979143 1784 tablet_service.cc:1467] Processing CreateTablet for tablet f696e1a2fda44d9fae364f65d666fd76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a85f01a53e964ccd87295fd3e83cd677]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:19:34.979472 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f696e1a2fda44d9fae364f65d666fd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:34.979800 2013 tablet_bootstrap.cc:654] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.981215 2018 tablet_bootstrap.cc:492] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: Bootstrap starting.
I20260620 12:19:34.981701 2014 tablet_bootstrap.cc:492] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: Bootstrap starting.
I20260620 12:19:34.982151 2018 tablet_bootstrap.cc:654] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.982400 2013 tablet_bootstrap.cc:492] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: No bootstrap required, opened a new log
I20260620 12:19:34.982497 2013 ts_tablet_manager.cc:1397] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 12:19:34.982542 2014 tablet_bootstrap.cc:654] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:34.983023 2013 raft_consensus.cc:348] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.983175 2013 raft_consensus.cc:374] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.983228 2013 raft_consensus.cc:729] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9543f151f1e416f959ac206a5e6adcf, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.983374 2018 tablet_bootstrap.cc:492] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: No bootstrap required, opened a new log
I20260620 12:19:34.983374 2013 consensus_queue.cc:260] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.983459 2018 ts_tablet_manager.cc:1397] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 12:19:34.983600 2014 tablet_bootstrap.cc:492] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: No bootstrap required, opened a new log
I20260620 12:19:34.983681 2013 ts_tablet_manager.cc:1428] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:19:34.983692 2014 ts_tablet_manager.cc:1397] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:34.983886 2018 raft_consensus.cc:348] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.983994 2018 raft_consensus.cc:374] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.984033 2018 raft_consensus.cc:729] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0675d73fdbeb494d83896c434f103587, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.984062 2014 raft_consensus.cc:348] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.984167 2014 raft_consensus.cc:374] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:34.984205 2014 raft_consensus.cc:729] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ebcb049b1146f29203690b26c74fa6, State: Initialized, Role: FOLLOWER
I20260620 12:19:34.984171 2018 consensus_queue.cc:260] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.984297 2017 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:34.984329 2014 consensus_queue.cc:260] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:34.984419 2017 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:34.984448 2018 ts_tablet_manager.cc:1428] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:34.984575 2014 ts_tablet_manager.cc:1428] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:34.984623 2021 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:34.984736 2021 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:34.984771 2023 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:34.984898 2023 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:35.238932 2023 raft_consensus.cc:482] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:35.239078 2023 raft_consensus.cc:504] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:35.239461 2023 leader_election.cc:288] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319), d9543f151f1e416f959ac206a5e6adcf (127.0.98.3:40811)
I20260620 12:19:35.240002 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f696e1a2fda44d9fae364f65d666fd76" candidate_uuid: "32ebcb049b1146f29203690b26c74fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0675d73fdbeb494d83896c434f103587" is_pre_election: true
I20260620 12:19:35.240042 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f696e1a2fda44d9fae364f65d666fd76" candidate_uuid: "32ebcb049b1146f29203690b26c74fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9543f151f1e416f959ac206a5e6adcf" is_pre_election: true
I20260620 12:19:35.240152 1664 raft_consensus.cc:2454] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ebcb049b1146f29203690b26c74fa6 in term 0.
I20260620 12:19:35.240232 1944 raft_consensus.cc:2454] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32ebcb049b1146f29203690b26c74fa6 in term 0.
I20260620 12:19:35.240597 1736 leader_election.cc:302] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0675d73fdbeb494d83896c434f103587, 32ebcb049b1146f29203690b26c74fa6, d9543f151f1e416f959ac206a5e6adcf; no voters:
I20260620 12:19:35.240840 2023 raft_consensus.cc:2781] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:35.240918 2023 raft_consensus.cc:482] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:35.240973 2023 raft_consensus.cc:3037] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:35.242079 2023 raft_consensus.cc:504] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:35.242427 2023 leader_election.cc:288] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [CANDIDATE]: Term 1 election: Requested vote from peers 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319), d9543f151f1e416f959ac206a5e6adcf (127.0.98.3:40811)
I20260620 12:19:35.242772 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f696e1a2fda44d9fae364f65d666fd76" candidate_uuid: "32ebcb049b1146f29203690b26c74fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0675d73fdbeb494d83896c434f103587"
I20260620 12:19:35.242774 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f696e1a2fda44d9fae364f65d666fd76" candidate_uuid: "32ebcb049b1146f29203690b26c74fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9543f151f1e416f959ac206a5e6adcf"
I20260620 12:19:35.242899 1664 raft_consensus.cc:3037] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:35.242900 1944 raft_consensus.cc:3037] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:35.243835 1944 raft_consensus.cc:2454] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ebcb049b1146f29203690b26c74fa6 in term 1.
I20260620 12:19:35.244021 1664 raft_consensus.cc:2454] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32ebcb049b1146f29203690b26c74fa6 in term 1.
I20260620 12:19:35.244177 1738 leader_election.cc:302] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ebcb049b1146f29203690b26c74fa6, d9543f151f1e416f959ac206a5e6adcf; no voters:
I20260620 12:19:35.244366 2023 raft_consensus.cc:2781] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:35.244494 2023 raft_consensus.cc:686] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 32ebcb049b1146f29203690b26c74fa6, State: Running, Role: LEADER
I20260620 12:19:35.244635 2023 consensus_queue.cc:237] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } }
I20260620 12:19:35.245275 2019 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: TxnStatusTablet state changed. Reason: New leader 32ebcb049b1146f29203690b26c74fa6. Latest consensus state: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:35.245384 2019 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:19:35.245589 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:19:35.245747 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:19:35.246374 1507 catalog_manager.cc:5495] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32ebcb049b1146f29203690b26c74fa6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } health_report { overall_health: UNKNOWN } } }
I20260620 12:19:35.273936 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.270266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35747 (local address 127.0.0.1:57334)
0620 12:19:35.270391 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:35.270406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:35.270509 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:35.270801 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:35.270803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:35.270812 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:19:35.270899 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:35.270903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.271870 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.271873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:35.272562 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:35.272572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.273159 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.273162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:35.273277 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.273354 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:19:35.273682 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:19:35.273695 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 12:19:35.273754 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 12:19:35.273821 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 12:19:35.273965 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.270363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.0.98.2:35747)
0620 12:19:35.270488 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:35.270492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:35.270508 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:19:35.270630 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:35.270633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:35.270698 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:35.270781 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:35.270995 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.271776 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.272673 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.273017 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.273145 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.273487 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:35.273568 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:35.273639 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:35.273786 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 12:19:35.273837 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:19:35.276602 1664 raft_consensus.cc:1261] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Refusing update from remote peer 32ebcb049b1146f29203690b26c74fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:35.276613 1944 raft_consensus.cc:1261] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Refusing update from remote peer 32ebcb049b1146f29203690b26c74fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:35.276998 2023 consensus_queue.cc:1035] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:35.277230 2019 consensus_queue.cc:1035] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:35.278604 2021 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: TxnStatusTablet state changed. Reason: New leader 32ebcb049b1146f29203690b26c74fa6. Latest consensus state: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:35.278720 2021 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:35.279699 2017 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: TxnStatusTablet state changed. Reason: New leader 32ebcb049b1146f29203690b26c74fa6. Latest consensus state: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:35.279810 2017 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:35.281028 2023 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:35.281168 2023 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:19:35.281456 2023 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:35.281545 2023 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P 32ebcb049b1146f29203690b26c74fa6: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:19:35.282081 2017 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:35.282178 2017 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P d9543f151f1e416f959ac206a5e6adcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:35.282629 2021 tablet_replica.cc:439] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32ebcb049b1146f29203690b26c74fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 } } peers { permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 } } peers { permanent_uuid: "d9543f151f1e416f959ac206a5e6adcf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40811 } } }
I20260620 12:19:35.282732 2021 tablet_replica.cc:442] T f696e1a2fda44d9fae364f65d666fd76 P 0675d73fdbeb494d83896c434f103587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:35.290923 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.286965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.0.98.3:40811)
0620 12:19:35.287189 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:35.287194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:35.287269 (+ 75us) server_negotiation.cc:407] Connection header received
0620 12:19:35.287404 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:35.287407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:35.287464 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:35.287582 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:35.287731 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.288494 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.289370 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.289714 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.289811 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.290028 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:35.290547 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:35.290657 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:35.290716 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 12:19:35.290782 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260620 12:19:35.300729 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.297105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35747 (local address 127.0.98.3:45773)
0620 12:19:35.297388 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:35.297400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:35.297489 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:35.297782 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:35.297784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:35.297791 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:19:35.297875 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:35.297880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.298862 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.298865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:35.299513 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:35.299522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.300097 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.300100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:35.300183 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.300222 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:19:35.300506 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:19:35.300510 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:19:35.300556 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 12:19:35.300616 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":114,"threads_started":1}
I20260620 12:19:35.300740 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.297259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45773 (local address 127.0.98.2:35747)
0620 12:19:35.297408 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:35.297413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:35.297478 (+ 65us) server_negotiation.cc:407] Connection header received
0620 12:19:35.297611 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:35.297614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:35.297666 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:35.297747 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:35.297975 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.298746 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.299631 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.299965 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.300076 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.300353 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:35.300427 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:35.300495 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:35.300566 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 12:19:35.300617 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:19:35.307535 1664 raft_consensus.cc:1261] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Refusing update from remote peer d9543f151f1e416f959ac206a5e6adcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:35.307775 1804 raft_consensus.cc:1261] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Refusing update from remote peer d9543f151f1e416f959ac206a5e6adcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:35.307991 2042 consensus_queue.cc:1035] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0675d73fdbeb494d83896c434f103587" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:35.308758 2042 consensus_queue.cc:1035] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ebcb049b1146f29203690b26c74fa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:35.312137 2026 mvcc.cc:205] Tried to move back new op lower bound from 7298899866855796736 to 7298899865236254720. Current Snapshot: MvccSnapshot[applied={T|T < 7298899866855796736}]
I20260620 12:19:35.329120 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.325532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40811 (local address 127.0.98.2:50255)
0620 12:19:35.325782 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:35.325799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:35.325921 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:35.326192 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:35.326195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:35.326203 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:19:35.326315 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:35.326321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.327229 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.327236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:35.327931 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:35.327940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.328383 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.328387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:35.328472 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.328517 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:19:35.328811 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:19:35.328820 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:19:35.328880 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 12:19:35.328960 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260620 12:19:35.329098 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.325577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50255 (local address 127.0.98.3:40811)
0620 12:19:35.325699 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:35.325703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:35.325872 (+ 169us) server_negotiation.cc:407] Connection header received
0620 12:19:35.326010 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:35.326013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:35.326087 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:35.326164 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:35.326408 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.327127 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.328043 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.328276 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.328386 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.328650 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:35.328726 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:35.328807 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:35.328898 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 12:19:35.328942 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:19:35.355592 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:35.355760 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:35.357978 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 032d23a61d5b45afa0bdde126cefd2ae: Sending DeleteTablet for 3 replicas of tablet 17ef64ea250b4527984f489d697fb725
I20260620 12:19:35.358656 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ef64ea250b4527984f489d697fb725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36618
I20260620 12:19:35.358862 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ef64ea250b4527984f489d697fb725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43134
I20260620 12:19:35.359061 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ef64ea250b4527984f489d697fb725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57320
I20260620 12:19:35.359404 2071 tablet_replica.cc:330] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: stopping tablet replica
I20260620 12:19:35.359529 2071 raft_consensus.cc:2229] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:35.359642 2071 raft_consensus.cc:2258] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:35.359836 2073 tablet_replica.cc:330] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: stopping tablet replica
I20260620 12:19:35.359933 2073 raft_consensus.cc:2229] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:35.360026 2073 raft_consensus.cc:2258] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:35.360680 2073 ts_tablet_manager.cc:1905] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:35.360728 2072 tablet_replica.cc:330] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: stopping tablet replica
I20260620 12:19:35.360932 2072 raft_consensus.cc:2229] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:19:35.361318 2072 raft_consensus.cc:2258] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:35.362490 2072 ts_tablet_manager.cc:1905] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:35.363545 2073 ts_tablet_manager.cc:1918] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:35.363627 2073 log.cc:1196] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/17ef64ea250b4527984f489d697fb725
I20260620 12:19:35.363888 2073 ts_tablet_manager.cc:1939] T 17ef64ea250b4527984f489d697fb725 P 32ebcb049b1146f29203690b26c74fa6: Deleting consensus metadata
I20260620 12:19:35.364552 1492 catalog_manager.cc:4841] TS 32ebcb049b1146f29203690b26c74fa6 (127.0.98.2:35747): tablet 17ef64ea250b4527984f489d697fb725 (table test-table [id=cecd1942143e4898b3f9773a92da0e15]) successfully deleted
I20260620 12:19:35.365275 2071 ts_tablet_manager.cc:1905] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:35.365962 2072 ts_tablet_manager.cc:1918] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:35.366092 2072 log.cc:1196] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/17ef64ea250b4527984f489d697fb725
I20260620 12:19:35.366449 2072 ts_tablet_manager.cc:1939] T 17ef64ea250b4527984f489d697fb725 P d9543f151f1e416f959ac206a5e6adcf: Deleting consensus metadata
I20260620 12:19:35.367194 1494 catalog_manager.cc:4841] TS d9543f151f1e416f959ac206a5e6adcf (127.0.98.3:40811): tablet 17ef64ea250b4527984f489d697fb725 (table test-table [id=cecd1942143e4898b3f9773a92da0e15]) successfully deleted
I20260620 12:19:35.367954 2071 ts_tablet_manager.cc:1918] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:35.368024 2071 log.cc:1196] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/17ef64ea250b4527984f489d697fb725
I20260620 12:19:35.368265 2071 ts_tablet_manager.cc:1939] T 17ef64ea250b4527984f489d697fb725 P 0675d73fdbeb494d83896c434f103587: Deleting consensus metadata
I20260620 12:19:35.368848 1492 catalog_manager.cc:4841] TS 0675d73fdbeb494d83896c434f103587 (127.0.98.1:39319): tablet 17ef64ea250b4527984f489d697fb725 (table test-table [id=cecd1942143e4898b3f9773a92da0e15]) successfully deleted
Jun 20 12:19:35 dist-test-slave-9b3m krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957974, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:35.385510 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.361751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.0.98.1:39319)
0620 12:19:35.361868 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:35.361873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:35.372411 (+ 10538us) server_negotiation.cc:407] Connection header received
0620 12:19:35.373309 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:35.373313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:35.373369 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:35.373424 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:35.374557 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.375369 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.376561 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.376891 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.376970 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.381014 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:35.381028 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:35.381077 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:35.381108 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:35.383293 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:35.383825 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:35.383832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:35.383834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:35.383925 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:35.384278 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:35.384282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:35.384284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:35.384577 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:35.384747 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:35.385196 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 12:19:35.385323 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 12:19:35.386457 1691 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36632
I20260620 12:19:35.402211 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:35.389475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.0.98.62:34727)
0620 12:19:35.389608 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:35.389612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:35.393338 (+ 3726us) server_negotiation.cc:407] Connection header received
0620 12:19:35.393524 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:35.393527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:35.393578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:35.393630 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:35.394501 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.395298 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.396425 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:35.396789 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:35.396898 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:35.398701 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:35.398705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:35.398708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:35.398734 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:35.400440 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:35.400913 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:35.400916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:35.400918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:35.401003 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:35.401340 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:35.401343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:35.401344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:35.401554 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:35.401680 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:35.401922 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 12:19:35.402036 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 12:19:35.403147 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48720
I20260620 12:19:35.405001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 1579
I20260620 12:19:35.414417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 1719
I20260620 12:19:35.421614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 1859
I20260620 12:19:35.427686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 1475
2026-06-20T12:19:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3078 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:35 dist-test-slave-9b3m krb5kdc[2080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:35 dist-test-slave-9b3m krb5kdc[2080](info): set up 2 sockets
Jun 20 12:19:35 dist-test-slave-9b3m krb5kdc[2080](info): commencing operation
krb5kdc: starting...
W20260620 12:19:36.464916 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:36 dist-test-slave-9b3m krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:36.583956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:36.692320 2093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:36.692572 2093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:36.692615 2093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:36.696661 2093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:36.696725 2093 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:19:36.696746 2093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:36.696767 2093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:36.696827 2093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:36.696864 2093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:36.701999 2093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39331
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:36.703006 2093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:36.704021 2093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:36.712440 2098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:36.712486 2093 server_base.cc:1034] running on GCE node
W20260620 12:19:36.712440 2099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:36.712471 2101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:36.713251 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:36.714339 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:36.715534 2093 hybrid_clock.cc:648] HybridClock initialized: now 1781957976715514 us; error 30 us; skew 500 ppm
Jun 20 12:19:36 dist-test-slave-9b3m krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:36.718647 2093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:36.719532 2093 webserver.cc:460] Webserver started at http://127.0.98.62:44161/ using document root <none> and password file <none>
I20260620 12:19:36.720117 2093 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:36.720222 2093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:36.720475 2093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:36.722620 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "5c36bc4d33994fd28dd40c6bdf4a577a"
format_stamp: "Formatted at 2026-06-20 12:19:36 on dist-test-slave-9b3m"
server_key: "79f8c10fa17ed017b2a9e23adb6887c9"
server_key_iv: "627307126f1c15bef99d3268d169a58d"
server_key_version: "encryptionkey@0"
I20260620 12:19:36.723210 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c36bc4d33994fd28dd40c6bdf4a577a"
format_stamp: "Formatted at 2026-06-20 12:19:36 on dist-test-slave-9b3m"
server_key: "79f8c10fa17ed017b2a9e23adb6887c9"
server_key_iv: "627307126f1c15bef99d3268d169a58d"
server_key_version: "encryptionkey@0"
I20260620 12:19:36.727044 2093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:19:36.729367 2108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:36.730552 2093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:19:36.730692 2093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "5c36bc4d33994fd28dd40c6bdf4a577a"
format_stamp: "Formatted at 2026-06-20 12:19:36 on dist-test-slave-9b3m"
server_key: "79f8c10fa17ed017b2a9e23adb6887c9"
server_key_iv: "627307126f1c15bef99d3268d169a58d"
server_key_version: "encryptionkey@0"
I20260620 12:19:36.730803 2093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:36.751297 2093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:36.754806 2093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:36.755028 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:36.763209 2093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39331
I20260620 12:19:36.763212 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39331 every 8 connection(s)
I20260620 12:19:36.764168 2093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:36.766871 2171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:36.769549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 2093
I20260620 12:19:36.769670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:36.769949 392 external_mini_cluster.cc:1428] Setting key 53d2eb258b54fa3d9883c810f142ade3
I20260620 12:19:36.772794 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: Bootstrap starting.
I20260620 12:19:36.775305 2171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:36.776167 2171 log.cc:824] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:36.778541 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: No bootstrap required, opened a new log
Jun 20 12:19:36 dist-test-slave-9b3m krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:36.782697 2171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c36bc4d33994fd28dd40c6bdf4a577a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260620 12:19:36.783039 2171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:36.783123 2171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c36bc4d33994fd28dd40c6bdf4a577a, State: Initialized, Role: FOLLOWER
I20260620 12:19:36.783650 2171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c36bc4d33994fd28dd40c6bdf4a577a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260620 12:19:36.783855 2171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:36.783948 2171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:36.784062 2171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:36.784817 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:36.771341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.0.98.62:39331)
0620 12:19:36.771754 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:36.771768 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:36.771802 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:19:36.772392 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:36.772408 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:36.772754 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:36.773126 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:36.773952 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:36.774940 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:36.776153 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:36.776801 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:36.776974 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:36.780021 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:36.780026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:36.780038 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:36.780064 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:36.781811 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:36.782289 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:36.782293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:36.782298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:36.782411 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:36.782692 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:36.782694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:36.782695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:36.783241 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:36.783478 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:36.783725 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 12:19:36.784076 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260620 12:19:36.784920 2171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c36bc4d33994fd28dd40c6bdf4a577a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260620 12:19:36.785277 2171 leader_election.cc:302] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c36bc4d33994fd28dd40c6bdf4a577a; no voters:
I20260620 12:19:36.785597 2171 leader_election.cc:288] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:36.785684 2176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:36.786794 2171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:36.786983 2176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [term 1 LEADER]: Becoming Leader. State: Replica: 5c36bc4d33994fd28dd40c6bdf4a577a, State: Running, Role: LEADER
I20260620 12:19:36.787339 2176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c36bc4d33994fd28dd40c6bdf4a577a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260620 12:19:36.788946 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c36bc4d33994fd28dd40c6bdf4a577a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c36bc4d33994fd28dd40c6bdf4a577a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } } }
I20260620 12:19:36.789232 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:36.789028 2178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c36bc4d33994fd28dd40c6bdf4a577a. Latest consensus state: current_term: 1 leader_uuid: "5c36bc4d33994fd28dd40c6bdf4a577a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c36bc4d33994fd28dd40c6bdf4a577a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } } }
I20260620 12:19:36.789528 2178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a [sys.catalog]: This master's current role is: LEADER
W20260620 12:19:36.791731 2190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:19:36.791807 2190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:19:36.791886 2191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:36.792555 2191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:36.797859 2191 catalog_manager.cc:1347] Generated new cluster ID: 90b0b948673d4ac5aba64a6d928fec86
I20260620 12:19:36.797926 2191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:36.803390 2191 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:36.804502 2191 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:36.811645 2191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: Generated new TSK 0
I20260620 12:19:36.812224 2191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:36.848430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:36.973024 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:36.973284 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:36.973373 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:36.977334 2197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:36.977389 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:36.977474 2197 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:19:36.977505 2197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:19:36.977588 2197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:36.982677 2197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:36.983690 2197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:36.984714 2197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:36.994503 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:36.994518 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:36.994509 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:36.994582 2197 server_base.cc:1034] running on GCE node
I20260620 12:19:36.995177 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:36.995792 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:36.996979 2197 hybrid_clock.cc:648] HybridClock initialized: now 1781957976996964 us; error 30 us; skew 500 ppm
Jun 20 12:19:36 dist-test-slave-9b3m krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:36.999652 2197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:37.000550 2197 webserver.cc:460] Webserver started at http://127.0.98.1:33197/ using document root <none> and password file <none>
I20260620 12:19:37.001160 2197 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:37.001250 2197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:37.001525 2197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:37.003523 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c1dd6cae7d7422e999b1b33cda56666"
format_stamp: "Formatted at 2026-06-20 12:19:36 on dist-test-slave-9b3m"
server_key: "87e47fe951298c8eb0b813840ab78a59"
server_key_iv: "609f6c4cbfb0f43fb4858a4c343f1bb2"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.004076 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c1dd6cae7d7422e999b1b33cda56666"
format_stamp: "Formatted at 2026-06-20 12:19:36 on dist-test-slave-9b3m"
server_key: "87e47fe951298c8eb0b813840ab78a59"
server_key_iv: "609f6c4cbfb0f43fb4858a4c343f1bb2"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.007766 2197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:19:37.010193 2212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:37.011353 2197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:37.011492 2197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "4c1dd6cae7d7422e999b1b33cda56666"
format_stamp: "Formatted at 2026-06-20 12:19:36 on dist-test-slave-9b3m"
server_key: "87e47fe951298c8eb0b813840ab78a59"
server_key_iv: "609f6c4cbfb0f43fb4858a4c343f1bb2"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.011598 2197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:37.022051 2197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:37.024775 2197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:37.024967 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:37.026353 2197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:37.026428 2197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:37.026526 2197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:37.026580 2197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:19:37 dist-test-slave-9b3m krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:37.037773 2197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34143
I20260620 12:19:37.038491 2327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34143 every 8 connection(s)
I20260620 12:19:37.038702 2197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:37.040345 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.027335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50211 (local address 127.0.98.62:39331)
0620 12:19:37.027458 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.027462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.028498 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 12:19:37.029673 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.029679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.029747 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.029849 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:37.031312 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.031847 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.032992 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.033303 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.033421 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.036565 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:37.036570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:37.036573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:37.036603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:37.038323 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.038883 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.038886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.038888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.038996 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.039415 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.039419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.039424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.039623 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:37.039700 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.040117 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.040226 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:19:37.041208 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.027862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.1:50211)
0620 12:19:37.028339 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.028378 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.029406 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.030091 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.030099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.030543 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:37.031154 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.031177 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.032005 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.032009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.032861 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.032869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.033499 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.033506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.033729 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.034357 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:37.034379 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:37.036361 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:37.038439 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.038444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.038458 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.038788 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.039122 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.039125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.039127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.039285 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.039698 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:37.039703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:37.039908 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.040066 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.040296 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":134,"threads_started":1}
I20260620 12:19:37.042343 2328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260620 12:19:37.042470 2328 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:37.043030 2328 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260620 12:19:37.044512 2124 ts_manager.cc:194] Registered new tserver with Master: 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143)
I20260620 12:19:37.044674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 2197
I20260620 12:19:37.044792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:37.045023 392 external_mini_cluster.cc:1428] Setting key adce55c37b03a6a49a9239ae209da073
I20260620 12:19:37.045970 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50211
I20260620 12:19:37.052147 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.044607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48967 (local address 127.0.98.62:39331)
0620 12:19:37.045188 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.045192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.045311 (+ 119us) server_negotiation.cc:407] Connection header received
0620 12:19:37.045478 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.045481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.045528 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.045618 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:37.046127 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.046840 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.047696 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.047918 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.048026 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.049240 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:37.049243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:37.049246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:37.049269 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:37.050589 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.051001 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.051007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.051009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.051089 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.051360 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.051365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.051367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.051539 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:37.051645 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.051891 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.051992 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511}
I20260620 12:19:37.052174 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.044527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.1:48967)
0620 12:19:37.045229 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.045245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.045376 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.045762 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.045764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.045916 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:37.046015 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.046020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.046997 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.047000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.047556 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.047564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.048027 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.048030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.048119 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.048629 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:37.048641 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:37.049106 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:37.050699 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.050704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.050706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.050898 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.051185 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.051188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.051190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.051272 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.051635 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:37.051638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:37.051735 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.051842 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.051989 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:37.074298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:37.183939 2337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:37.184177 2337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:37.184222 2337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:37.188190 2337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:37.188246 2337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:37.188320 2337 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:19:37.188362 2337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:19:37.188397 2337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:37.193531 2337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:37.194418 2337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:37.195324 2337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:37.204708 2342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:37.204833 2343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:37.204943 2337 server_base.cc:1034] running on GCE node
W20260620 12:19:37.204708 2345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:37.205356 2337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:37.205932 2337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:37.207130 2337 hybrid_clock.cc:648] HybridClock initialized: now 1781957977207116 us; error 27 us; skew 500 ppm
Jun 20 12:19:37 dist-test-slave-9b3m krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:37.210080 2337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:37.210913 2337 webserver.cc:460] Webserver started at http://127.0.98.2:45513/ using document root <none> and password file <none>
I20260620 12:19:37.211458 2337 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:37.211549 2337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:37.211781 2337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:37.213893 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "8fa347d18f3a4824b04ab1f58e3d7952"
format_stamp: "Formatted at 2026-06-20 12:19:37 on dist-test-slave-9b3m"
server_key: "de3ada087b07743ca396e0e1faa5f717"
server_key_iv: "7396b9434acb4481d38ace53eb2b743d"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.214437 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8fa347d18f3a4824b04ab1f58e3d7952"
format_stamp: "Formatted at 2026-06-20 12:19:37 on dist-test-slave-9b3m"
server_key: "de3ada087b07743ca396e0e1faa5f717"
server_key_iv: "7396b9434acb4481d38ace53eb2b743d"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.218168 2337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 12:19:37.220455 2352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:37.221665 2337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:37.221797 2337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "8fa347d18f3a4824b04ab1f58e3d7952"
format_stamp: "Formatted at 2026-06-20 12:19:37 on dist-test-slave-9b3m"
server_key: "de3ada087b07743ca396e0e1faa5f717"
server_key_iv: "7396b9434acb4481d38ace53eb2b743d"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.221903 2337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:37.232538 2337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:37.235653 2337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:37.235836 2337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:37.237260 2337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:37.237329 2337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:37.237402 2337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:37.237449 2337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:19:37 dist-test-slave-9b3m krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:37.247993 2337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45725
I20260620 12:19:37.248063 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45725 every 8 connection(s)
I20260620 12:19:37.248805 2337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:37.250099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 2337
I20260620 12:19:37.250224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:37.250523 392 external_mini_cluster.cc:1428] Setting key f410f022512d5e1689bccacbd08fdd3d
I20260620 12:19:37.251618 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.238352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.2:53345)
0620 12:19:37.238774 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.238822 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.239819 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.240653 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.240661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.241140 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:37.241659 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.241671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.242498 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.242501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.243607 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.243616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.244340 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.244346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.244566 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.245176 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:37.245200 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:37.246664 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:37.248531 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.248539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.248556 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.248813 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.249157 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.249160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.249162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.249331 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.249801 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:37.249806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:37.250113 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.250331 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.250671 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
I20260620 12:19:37.252249 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.237950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53345 (local address 127.0.98.62:39331)
0620 12:19:37.238087 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.238092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.238945 (+ 853us) server_negotiation.cc:407] Connection header received
0620 12:19:37.240067 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.240072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.240136 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.240200 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:37.241799 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.242372 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.243760 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.244165 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.244302 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.246830 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:37.246833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:37.246836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:37.246868 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:37.248420 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.248916 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.248921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.248922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.249006 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.249457 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.249461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.249462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.249677 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:37.249798 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.251988 (+ 2190us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.252145 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:19:37.253242 2468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260620 12:19:37.253407 2468 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:37.253997 2468 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260620 12:19:37.255026 2124 ts_manager.cc:194] Registered new tserver with Master: 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2:45725)
I20260620 12:19:37.255690 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53345
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:19:37.262661 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.255265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.2:54293)
0620 12:19:37.255535 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.255549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.255629 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.255938 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.255941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.256142 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:37.256256 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.256265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.257424 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.257427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.258040 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.258049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.258617 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.258619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.258699 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.259204 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:37.259215 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:37.259505 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:37.261192 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.261195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.261196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.261373 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.261672 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.261677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.261679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.261758 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.262153 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:37.262156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:37.262245 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.262348 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.262486 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260620 12:19:37.262710 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.255576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54293 (local address 127.0.98.62:39331)
0620 12:19:37.255694 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.255700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.255715 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:19:37.255753 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.255759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.255810 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.255897 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:37.256382 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.257269 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.258160 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.258504 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.258623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.259640 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:37.259644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:37.259646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:37.259673 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:37.261038 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.261490 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.261496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.261498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.261579 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.261850 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.261855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.261857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.262056 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:37.262164 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.262410 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.262579 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:19:37.277493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:37.384418 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:37.384660 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:37.384706 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:37.388881 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:37.388935 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:37.389032 2477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 12:19:37.389072 2477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:19:37.389106 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:37.394137 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:37.395037 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:37.395936 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:37.404994 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:37.405097 2477 server_base.cc:1034] running on GCE node
W20260620 12:19:37.405011 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:37.405006 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:37.405670 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:37.406245 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:37.407406 2477 hybrid_clock.cc:648] HybridClock initialized: now 1781957977407392 us; error 29 us; skew 500 ppm
Jun 20 12:19:37 dist-test-slave-9b3m krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:37.410095 2477 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:37.410907 2477 webserver.cc:460] Webserver started at http://127.0.98.3:45007/ using document root <none> and password file <none>
I20260620 12:19:37.411448 2477 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:37.411535 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:37.411748 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:37.413803 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ba19504f8764362b7e9e6ea65d8885c"
format_stamp: "Formatted at 2026-06-20 12:19:37 on dist-test-slave-9b3m"
server_key: "88351ad48288b0857c5173473c2b58a2"
server_key_iv: "9af81b3481f15a857ee2d2ce5fdca602"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.414342 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ba19504f8764362b7e9e6ea65d8885c"
format_stamp: "Formatted at 2026-06-20 12:19:37 on dist-test-slave-9b3m"
server_key: "88351ad48288b0857c5173473c2b58a2"
server_key_iv: "9af81b3481f15a857ee2d2ce5fdca602"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.418010 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:37.420282 2492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:37.421548 2477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:37.421676 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "3ba19504f8764362b7e9e6ea65d8885c"
format_stamp: "Formatted at 2026-06-20 12:19:37 on dist-test-slave-9b3m"
server_key: "88351ad48288b0857c5173473c2b58a2"
server_key_iv: "9af81b3481f15a857ee2d2ce5fdca602"
server_key_version: "encryptionkey@0"
I20260620 12:19:37.421780 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:37.444402 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:37.447386 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:37.447602 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:37.449146 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:37.449210 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:37.449277 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:37.449323 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:19:37 dist-test-slave-9b3m krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:37.460350 2477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39705
I20260620 12:19:37.460393 2607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39705 every 8 connection(s)
I20260620 12:19:37.461508 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:37.463104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 2477
I20260620 12:19:37.463261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:37.463627 392 external_mini_cluster.cc:1428] Setting key a21f30fea8a29aaf567b596d16017288
I20260620 12:19:37.463706 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.450021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34241 (local address 127.0.98.62:39331)
0620 12:19:37.450172 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.450176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.451075 (+ 899us) server_negotiation.cc:407] Connection header received
0620 12:19:37.452414 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.452419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.452507 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.452624 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:37.454379 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.455140 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.456431 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.456672 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.456790 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.459416 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:37.459420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:37.459423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:37.459449 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:37.461234 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.461888 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.461892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.461894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.462004 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.462424 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.462427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.462429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.462617 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:37.462732 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.463365 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.463524 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:19:37.464540 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.450402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.3:34241)
0620 12:19:37.450877 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.450949 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.452150 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.452888 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.452902 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.453514 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:37.454211 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.454228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.455269 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.455273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.456311 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.456321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.456847 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.456852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.457096 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.457750 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:37.457766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:37.459239 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:37.461379 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.461384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.461396 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.461750 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.462118 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.462124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.462127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.462305 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.462728 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:37.462734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:37.463066 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.463286 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.463590 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260620 12:19:37.467739 2608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260620 12:19:37.467881 2608 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:37.468338 2608 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260620 12:19:37.469408 2124 ts_manager.cc:194] Registered new tserver with Master: 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:37.470021 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34241
I20260620 12:19:37.475054 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.468021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39839 (local address 127.0.98.62:39331)
0620 12:19:37.468156 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.468161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.468393 (+ 232us) server_negotiation.cc:407] Connection header received
0620 12:19:37.468543 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.468547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.468595 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.468689 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:37.469220 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.469795 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.470730 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.470961 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.471070 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.472121 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:37.472125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:37.472127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:37.472156 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:37.473500 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.473900 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.473905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.473907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.474003 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.474277 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.474282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.474284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.474446 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:37.474549 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.474799 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.474903 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:19:37.475063 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.467905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.3:39839)
0620 12:19:37.468296 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.468308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.468389 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.468703 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.468705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.468859 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:37.468954 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.468959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.469916 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.469919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.470616 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.470625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.471072 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.471076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.471169 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.471683 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:37.471694 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:37.471992 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:37.473610 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.473614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.473616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.473802 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.474098 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.474102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.474104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.474185 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.474550 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:37.474553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:37.474643 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.474747 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.474877 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":114,"threads_started":1}
I20260620 12:19:37.477952 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:19:37 dist-test-slave-9b3m krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:19:37 dist-test-slave-9b3m krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:37.503908 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.495811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.0.98.62:39331)
0620 12:19:37.495972 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.495975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.495987 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:19:37.496028 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.496031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.496069 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.496131 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:37.496797 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.497408 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.498263 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.498480 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.498559 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.500946 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:37.500949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:37.500951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:37.500976 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:37.502525 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.502900 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.502902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.502904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.502976 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.503203 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.503205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.503206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.503372 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:37.503475 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.503668 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.503760 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 12:19:37.506991 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:19:37.510031 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:37.524103 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.518027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45725 (local address 127.0.0.1:38286)
0620 12:19:37.518416 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.518470 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.518634 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.519221 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.519227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.519250 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:37.519513 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.519526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.521098 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.521105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.522153 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.522163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.523410 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.523446 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.523666 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.523712 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.523837 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.523952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":64,"thread_start_us":113,"threads_started":1}
I20260620 12:19:37.524125 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.518217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.0.98.2:45725)
0620 12:19:37.518571 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.518584 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.518610 (+ 26us) server_negotiation.cc:407] Connection header received
0620 12:19:37.518814 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.518820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.519044 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.519692 (+ 648us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:37.519857 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.520959 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.522295 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.523270 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.523397 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.523592 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.523889 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.523984 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":133,"threads_started":1}
I20260620 12:19:37.524889 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.518811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39705 (local address 127.0.0.1:41254)
0620 12:19:37.519064 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.519077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.519167 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.520289 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.520294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.520316 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:37.520430 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.520436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.522023 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.522028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.523360 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.523369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.524507 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.524513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.524621 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.524643 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.524695 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.524755 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":17,"thread_start_us":81,"threads_started":1}
I20260620 12:19:37.525568 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.519022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.0.98.3:39705)
0620 12:19:37.519825 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.519838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.519861 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:19:37.519939 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.519944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.520130 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.520310 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:37.520561 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.521795 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.523489 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.524365 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.525080 (+ 715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.525237 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.525328 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.525432 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":101,"threads_started":1}
I20260620 12:19:37.526144 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.519108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34143 (local address 127.0.0.1:47530)
0620 12:19:37.519450 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.519467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.519570 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.520468 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.520471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.520490 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:37.520578 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.520585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.522048 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.522050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.524541 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.524551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.525736 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.525742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.525847 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.525873 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.525946 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.525994 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":49,"thread_start_us":82,"threads_started":1}
I20260620 12:19:37.526228 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.519632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.0.98.1:34143)
0620 12:19:37.519985 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.519997 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.520016 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:19:37.520075 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.520078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.520216 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.520383 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:37.520707 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.521912 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.524740 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.525540 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.525637 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.525831 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.525991 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.526078 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":86,"threads_started":1}
I20260620 12:19:37.527587 2402 tablet_service.cc:1467] Processing CreateTablet for tablet e45e3c7e24604c6d9436cf774667154c (DEFAULT_TABLE table=test-table [id=e7f5305227514e06b3d5c331e5dc839f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:37.528704 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45e3c7e24604c6d9436cf774667154c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:37.528820 2542 tablet_service.cc:1467] Processing CreateTablet for tablet e45e3c7e24604c6d9436cf774667154c (DEFAULT_TABLE table=test-table [id=e7f5305227514e06b3d5c331e5dc839f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:37.529228 2262 tablet_service.cc:1467] Processing CreateTablet for tablet e45e3c7e24604c6d9436cf774667154c (DEFAULT_TABLE table=test-table [id=e7f5305227514e06b3d5c331e5dc839f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:37.529830 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45e3c7e24604c6d9436cf774667154c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:37.530162 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45e3c7e24604c6d9436cf774667154c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:37.537097 2631 tablet_bootstrap.cc:492] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: Bootstrap starting.
I20260620 12:19:37.538110 2633 tablet_bootstrap.cc:492] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Bootstrap starting.
I20260620 12:19:37.539409 2631 tablet_bootstrap.cc:654] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:37.540105 2633 tablet_bootstrap.cc:654] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:37.540180 2631 log.cc:824] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:37.540822 2633 log.cc:824] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:37.541366 2632 tablet_bootstrap.cc:492] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Bootstrap starting.
I20260620 12:19:37.542340 2631 tablet_bootstrap.cc:492] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: No bootstrap required, opened a new log
I20260620 12:19:37.542557 2631 ts_tablet_manager.cc:1397] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:19:37.542768 2633 tablet_bootstrap.cc:492] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: No bootstrap required, opened a new log
I20260620 12:19:37.542929 2633 ts_tablet_manager.cc:1397] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:19:37.543368 2632 tablet_bootstrap.cc:654] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:37.544282 2632 log.cc:824] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:37.546243 2632 tablet_bootstrap.cc:492] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: No bootstrap required, opened a new log
I20260620 12:19:37.546169 2631 raft_consensus.cc:348] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.546422 2632 ts_tablet_manager.cc:1397] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:19:37.546444 2631 raft_consensus.cc:374] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:37.546494 2631 raft_consensus.cc:729] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1dd6cae7d7422e999b1b33cda56666, State: Initialized, Role: FOLLOWER
I20260620 12:19:37.546914 2631 consensus_queue.cc:260] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.546851 2633 raft_consensus.cc:348] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.547081 2633 raft_consensus.cc:374] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:37.547130 2633 raft_consensus.cc:729] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba19504f8764362b7e9e6ea65d8885c, State: Initialized, Role: FOLLOWER
I20260620 12:19:37.547550 2633 consensus_queue.cc:260] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.547797 2631 ts_tablet_manager.cc:1428] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:19:37.548166 2608 heartbeater.cc:503] Master 127.0.98.62:39331 was elected leader, sending a full tablet report...
I20260620 12:19:37.548427 2633 ts_tablet_manager.cc:1428] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:19:37.549198 2328 heartbeater.cc:503] Master 127.0.98.62:39331 was elected leader, sending a full tablet report...
I20260620 12:19:37.550424 2632 raft_consensus.cc:348] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.550663 2632 raft_consensus.cc:374] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:37.550720 2632 raft_consensus.cc:729] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Initialized, Role: FOLLOWER
I20260620 12:19:37.551153 2632 consensus_queue.cc:260] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.551739 2468 heartbeater.cc:503] Master 127.0.98.62:39331 was elected leader, sending a full tablet report...
I20260620 12:19:37.552997 2632 ts_tablet_manager.cc:1428] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:19:37.681689 2639 raft_consensus.cc:482] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:37.681936 2639 raft_consensus.cc:504] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.683131 2639 leader_election.cc:288] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705), 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143)
I20260620 12:19:37.687208 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.683496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38121 (local address 127.0.98.3:39705)
0620 12:19:37.683639 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.683643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.683661 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:19:37.683716 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.683719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.683770 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.683855 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:37.684276 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.684940 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.686048 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.686644 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.686750 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.686834 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.687054 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.687110 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:19:37.687224 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.683341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39705 (local address 127.0.98.2:38121)
0620 12:19:37.683472 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.683487 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.683588 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.683988 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.683991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.684008 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:37.684153 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.684162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.685070 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.685074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.685924 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.685933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.686754 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.686756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.686872 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.686952 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.687030 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.687099 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 12:19:37.687916 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45e3c7e24604c6d9436cf774667154c" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c" is_pre_election: true
I20260620 12:19:37.688122 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.683765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57447 (local address 127.0.98.1:34143)
0620 12:19:37.683927 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.683932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.684120 (+ 188us) server_negotiation.cc:407] Connection header received
0620 12:19:37.684221 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.684226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.684280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.684343 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:37.684609 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.685176 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.686846 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.687487 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.687606 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.687665 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.687950 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.688019 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 12:19:37.688117 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.683674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34143 (local address 127.0.98.2:57447)
0620 12:19:37.684023 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.684034 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.684118 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.684377 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.684379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.684396 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:37.684500 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.684506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.685272 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.685275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.686734 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.686743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.687734 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.687737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.687806 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.687823 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.687929 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.687992 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":111,"threads_started":1}
I20260620 12:19:37.688261 2550 raft_consensus.cc:2454] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 0.
I20260620 12:19:37.688736 2354 leader_election.cc:302] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba19504f8764362b7e9e6ea65d8885c, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:37.688740 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45e3c7e24604c6d9436cf774667154c" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1dd6cae7d7422e999b1b33cda56666" is_pre_election: true
I20260620 12:19:37.688998 2639 raft_consensus.cc:2781] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:37.689044 2282 raft_consensus.cc:2454] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 0.
I20260620 12:19:37.689117 2639 raft_consensus.cc:482] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:37.689167 2639 raft_consensus.cc:3037] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:37.690217 2639 raft_consensus.cc:504] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.690500 2639 leader_election.cc:288] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 election: Requested vote from peers 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705), 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143)
I20260620 12:19:37.690845 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45e3c7e24604c6d9436cf774667154c" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c"
I20260620 12:19:37.690938 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45e3c7e24604c6d9436cf774667154c" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1dd6cae7d7422e999b1b33cda56666"
I20260620 12:19:37.690990 2550 raft_consensus.cc:3037] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:37.691072 2282 raft_consensus.cc:3037] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:37.692158 2282 raft_consensus.cc:2454] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 1.
I20260620 12:19:37.692265 2550 raft_consensus.cc:2454] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 1.
I20260620 12:19:37.692468 2354 leader_election.cc:302] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1dd6cae7d7422e999b1b33cda56666, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:37.692644 2639 raft_consensus.cc:2781] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:37.692859 2639 raft_consensus.cc:686] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Running, Role: LEADER
I20260620 12:19:37.693284 2639 consensus_queue.cc:237] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } }
I20260620 12:19:37.696426 2125 catalog_manager.cc:5495] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } health_report { overall_health: UNKNOWN } } }
W20260620 12:19:37.712532 2609 tablet.cc:2369] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 12:19:37 dist-test-slave-9b3m krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:37.746697 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.738550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.0.1:48228)
0620 12:19:37.738808 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:37.738821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:37.738941 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:37.739163 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:37.739165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:37.739538 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:37.739627 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.739632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.740608 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.740610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.741231 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:37.741239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.741635 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.741637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:37.741694 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.742290 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:37.742302 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:37.743769 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:37.745251 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.745258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.745262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.745482 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.745766 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:37.745768 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:37.745769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:37.745842 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:37.746215 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:37.746221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:37.746318 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 12:19:37.746418 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 12:19:37.746533 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260620 12:19:37.746712 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:37.738598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.0.98.62:39331)
0620 12:19:37.738729 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:37.738734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:37.738896 (+ 162us) server_negotiation.cc:407] Connection header received
0620 12:19:37.739021 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:37.739024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:37.739062 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:37.739129 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:37.739727 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.740497 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.741340 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:37.741542 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:37.741623 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:37.743888 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:37.743891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:37.743894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:37.743915 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:37.745159 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.745598 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.745601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.745602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.745681 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:37.745956 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:37.745958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:37.745959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:37.746120 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:37.746195 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:37.746466 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 12:19:37.746588 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":55}
W20260620 12:19:37.750010 2469 tablet.cc:2369] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:19:37.750306 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48228:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:19:37.751021 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:37.755833 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 0acc9871a04a4c5d860bc640c56cc5a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7431bcbf67754c78a6aaf284cfcd9c0e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:19:37.755808 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 0acc9871a04a4c5d860bc640c56cc5a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7431bcbf67754c78a6aaf284cfcd9c0e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:19:37.755831 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 0acc9871a04a4c5d860bc640c56cc5a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7431bcbf67754c78a6aaf284cfcd9c0e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:19:37.756469 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acc9871a04a4c5d860bc640c56cc5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:37.756475 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acc9871a04a4c5d860bc640c56cc5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:37.756475 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acc9871a04a4c5d860bc640c56cc5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:37.759018 2632 tablet_bootstrap.cc:492] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: Bootstrap starting.
I20260620 12:19:37.759261 2631 tablet_bootstrap.cc:492] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: Bootstrap starting.
I20260620 12:19:37.759331 2633 tablet_bootstrap.cc:492] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: Bootstrap starting.
I20260620 12:19:37.760026 2632 tablet_bootstrap.cc:654] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:37.760108 2631 tablet_bootstrap.cc:654] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:37.760324 2633 tablet_bootstrap.cc:654] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:37.761262 2631 tablet_bootstrap.cc:492] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: No bootstrap required, opened a new log
I20260620 12:19:37.761364 2631 ts_tablet_manager.cc:1397] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:37.761559 2633 tablet_bootstrap.cc:492] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: No bootstrap required, opened a new log
I20260620 12:19:37.761651 2633 ts_tablet_manager.cc:1397] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:37.761703 2632 tablet_bootstrap.cc:492] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: No bootstrap required, opened a new log
I20260620 12:19:37.761799 2632 ts_tablet_manager.cc:1397] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 12:19:37.761828 2631 raft_consensus.cc:348] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:37.761991 2631 raft_consensus.cc:374] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:37.762032 2631 raft_consensus.cc:729] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1dd6cae7d7422e999b1b33cda56666, State: Initialized, Role: FOLLOWER
I20260620 12:19:37.762123 2633 raft_consensus.cc:348] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:37.762161 2631 consensus_queue.cc:260] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:37.762240 2633 raft_consensus.cc:374] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:37.762208 2632 raft_consensus.cc:348] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:37.762282 2633 raft_consensus.cc:729] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba19504f8764362b7e9e6ea65d8885c, State: Initialized, Role: FOLLOWER
I20260620 12:19:37.762310 2632 raft_consensus.cc:374] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:37.762351 2632 raft_consensus.cc:729] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Initialized, Role: FOLLOWER
I20260620 12:19:37.762450 2631 ts_tablet_manager.cc:1428] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:37.762426 2633 consensus_queue.cc:260] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:37.762477 2632 consensus_queue.cc:260] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:37.762679 2633 ts_tablet_manager.cc:1428] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:37.762730 2632 ts_tablet_manager.cc:1428] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:19:37.762924 2638 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:37.763054 2638 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:37.763187 2639 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:37.763334 2639 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:37.765509 2637 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:37.765661 2637 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 12:19:37.789969 2329 tablet.cc:2369] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:19:38.041142 2639 raft_consensus.cc:482] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:38.041293 2639 raft_consensus.cc:504] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.041786 2639 leader_election.cc:288] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143), 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:38.042343 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc9871a04a4c5d860bc640c56cc5a2" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c" is_pre_election: true
I20260620 12:19:38.042493 2550 raft_consensus.cc:2454] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 0.
I20260620 12:19:38.042586 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc9871a04a4c5d860bc640c56cc5a2" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1dd6cae7d7422e999b1b33cda56666" is_pre_election: true
I20260620 12:19:38.042793 2354 leader_election.cc:302] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba19504f8764362b7e9e6ea65d8885c, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:38.042788 2282 raft_consensus.cc:2454] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 0.
I20260620 12:19:38.042963 2639 raft_consensus.cc:2781] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:38.043004 2639 raft_consensus.cc:482] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:38.043037 2639 raft_consensus.cc:3037] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:38.044240 2639 raft_consensus.cc:504] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.044546 2639 leader_election.cc:288] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143), 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:38.045125 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc9871a04a4c5d860bc640c56cc5a2" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1dd6cae7d7422e999b1b33cda56666"
I20260620 12:19:38.045218 2282 raft_consensus.cc:3037] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:38.045217 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc9871a04a4c5d860bc640c56cc5a2" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c"
I20260620 12:19:38.045380 2550 raft_consensus.cc:3037] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:38.046245 2282 raft_consensus.cc:2454] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 1.
I20260620 12:19:38.046664 2354 leader_election.cc:302] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1dd6cae7d7422e999b1b33cda56666, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:38.046847 2639 raft_consensus.cc:2781] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:38.046916 2639 raft_consensus.cc:686] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Running, Role: LEADER
I20260620 12:19:38.047034 2639 consensus_queue.cc:237] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.047297 2550 raft_consensus.cc:2454] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 1.
I20260620 12:19:38.047614 2639 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: TxnStatusTablet state changed. Reason: New leader 8fa347d18f3a4824b04ab1f58e3d7952. Latest consensus state: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:38.047708 2639 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:19:38.048324 2653 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:19:38.048547 2653 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:19:38.048946 2124 catalog_manager.cc:5495] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } health_report { overall_health: UNKNOWN } } }
I20260620 12:19:38.108325 2641 consensus_queue.cc:1035] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:38.114972 2657 consensus_queue.cc:1035] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:38.128912 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.124807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45725 (local address 127.0.0.1:38294)
0620 12:19:38.124940 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:38.124957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:38.125103 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:38.125497 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:38.125504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:38.125511 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:19:38.125615 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.125620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.126656 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.126658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.127246 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.127254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.127833 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.127836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.127913 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.127954 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.128677 (+ 723us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.128687 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 12:19:38.128736 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 12:19:38.128802 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 12:19:38.128952 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.124899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.0.98.2:45725)
0620 12:19:38.125184 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:38.125192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:38.125208 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:19:38.125279 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:38.125282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:38.125342 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:38.125461 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:38.125692 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.126553 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.127355 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.127725 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.127820 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.128097 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.128585 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.128697 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:38.128750 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 12:19:38.128802 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":108,"threads_started":1}
I20260620 12:19:38.133077 2282 raft_consensus.cc:1261] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Refusing update from remote peer 8fa347d18f3a4824b04ab1f58e3d7952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:38.133103 2550 raft_consensus.cc:1261] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Refusing update from remote peer 8fa347d18f3a4824b04ab1f58e3d7952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:38.133486 2657 consensus_queue.cc:1035] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:38.133633 2641 consensus_queue.cc:1035] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:38.136336 2637 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: TxnStatusTablet state changed. Reason: New leader 8fa347d18f3a4824b04ab1f58e3d7952. Latest consensus state: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:38.136451 2637 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:38.137470 2638 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: TxnStatusTablet state changed. Reason: New leader 8fa347d18f3a4824b04ab1f58e3d7952. Latest consensus state: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:38.137585 2638 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:38.138617 2641 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:38.138723 2641 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:19:38.138929 2656 mvcc.cc:205] Tried to move back new op lower bound from 7298899878428475392 to 7298899878081843200. Current Snapshot: MvccSnapshot[applied={T|T < 7298899878428475392}]
I20260620 12:19:38.139744 2637 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:38.139842 2637 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 4c1dd6cae7d7422e999b1b33cda56666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:38.140107 2660 mvcc.cc:205] Tried to move back new op lower bound from 7298899878428475392 to 7298899878081843200. Current Snapshot: MvccSnapshot[applied={T|T < 7298899878428475392}]
I20260620 12:19:38.141613 2639 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:38.141707 2639 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 8fa347d18f3a4824b04ab1f58e3d7952: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:19:38.143394 2638 tablet_replica.cc:439] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } } }
I20260620 12:19:38.143520 2638 tablet_replica.cc:442] T 0acc9871a04a4c5d860bc640c56cc5a2 P 3ba19504f8764362b7e9e6ea65d8885c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:19:38.149228 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.145955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.0.98.2:45725)
0620 12:19:38.146050 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:38.146054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:38.146069 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:19:38.146152 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:38.146155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:38.146208 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:38.146286 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:38.146554 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.147275 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.148169 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.148502 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.148611 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.148801 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.148886 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.148963 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:38.149056 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 12:19:38.149109 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 12:19:38.159621 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.156048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38115 (local address 127.0.98.2:45725)
0620 12:19:38.156170 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:38.156178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:38.156402 (+ 224us) server_negotiation.cc:407] Connection header received
0620 12:19:38.156520 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:38.156522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:38.156558 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:38.156618 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:38.156882 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.157410 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.158665 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.158860 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.158940 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.159265 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.159320 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.159380 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:38.159494 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 12:19:38.159530 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:19:38.159791 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.155976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45725 (local address 127.0.98.2:38115)
0620 12:19:38.156320 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:38.156334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:38.156434 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:38.156648 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:38.156652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:38.156660 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:19:38.156784 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.156789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.157521 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.157525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.158559 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.158569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.158981 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.158986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.159079 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.159127 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.159406 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.159412 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:19:38.159480 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:19:38.159560 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
I20260620 12:19:38.200650 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:38.200832 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:38.203155 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: Sending DeleteTablet for 3 replicas of tablet e45e3c7e24604c6d9436cf774667154c
I20260620 12:19:38.203804 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet e45e3c7e24604c6d9436cf774667154c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38286
I20260620 12:19:38.204226 2675 tablet_replica.cc:330] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: stopping tablet replica
I20260620 12:19:38.204246 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet e45e3c7e24604c6d9436cf774667154c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41254
I20260620 12:19:38.204494 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:19:38.204761 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet e45e3c7e24604c6d9436cf774667154c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47530
I20260620 12:19:38.204836 2675 raft_consensus.cc:2229] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 LEADER]: Raft consensus shutting down.
W20260620 12:19:38.204906 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:38.205179 2677 tablet_replica.cc:330] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: stopping tablet replica
I20260620 12:19:38.205410 2675 raft_consensus.cc:2258] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:38.205471 2677 raft_consensus.cc:2229] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:38.205652 2677 raft_consensus.cc:2258] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:38.206805 2677 ts_tablet_manager.cc:1905] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:38.206805 2675 ts_tablet_manager.cc:1905] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:38.209607 2262 tablet_service.cc:1467] Processing CreateTablet for tablet d0c96d4c2dd6494cabfdf34090b636dc (DEFAULT_TABLE table=test-table [id=0455224cbfd740e5a12e9ec241dfc5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:38.209720 2677 ts_tablet_manager.cc:1918] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:38.209803 2677 log.cc:1196] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/e45e3c7e24604c6d9436cf774667154c
I20260620 12:19:38.209874 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c96d4c2dd6494cabfdf34090b636dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:38.209656 2675 ts_tablet_manager.cc:1918] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:38.210048 2675 log.cc:1196] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/e45e3c7e24604c6d9436cf774667154c
I20260620 12:19:38.210139 2677 ts_tablet_manager.cc:1939] T e45e3c7e24604c6d9436cf774667154c P 4c1dd6cae7d7422e999b1b33cda56666: Deleting consensus metadata
I20260620 12:19:38.210436 2675 ts_tablet_manager.cc:1939] T e45e3c7e24604c6d9436cf774667154c P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting consensus metadata
I20260620 12:19:38.211431 2676 tablet_replica.cc:330] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: stopping tablet replica
I20260620 12:19:38.211719 2676 raft_consensus.cc:2229] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:38.211889 2631 tablet_bootstrap.cc:492] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: Bootstrap starting.
I20260620 12:19:38.211978 2676 raft_consensus.cc:2258] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:38.212064 2542 tablet_service.cc:1467] Processing CreateTablet for tablet d0c96d4c2dd6494cabfdf34090b636dc (DEFAULT_TABLE table=test-table [id=0455224cbfd740e5a12e9ec241dfc5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:38.212291 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c96d4c2dd6494cabfdf34090b636dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:38.212780 2631 tablet_bootstrap.cc:654] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:38.213256 2110 catalog_manager.cc:4841] TS 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143): tablet e45e3c7e24604c6d9436cf774667154c (table test-table [id=e7f5305227514e06b3d5c331e5dc839f]) successfully deleted
I20260620 12:19:38.213307 2676 ts_tablet_manager.cc:1905] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:38.215236 2633 tablet_bootstrap.cc:492] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: Bootstrap starting.
I20260620 12:19:38.215706 2110 catalog_manager.cc:4841] TS 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2:45725): tablet e45e3c7e24604c6d9436cf774667154c (table test-table [id=e7f5305227514e06b3d5c331e5dc839f]) successfully deleted
I20260620 12:19:38.215873 2676 ts_tablet_manager.cc:1918] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:38.215950 2676 log.cc:1196] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/e45e3c7e24604c6d9436cf774667154c
I20260620 12:19:38.216020 2401 tablet_service.cc:1467] Processing CreateTablet for tablet d0c96d4c2dd6494cabfdf34090b636dc (DEFAULT_TABLE table=test-table [id=0455224cbfd740e5a12e9ec241dfc5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:38.216233 2633 tablet_bootstrap.cc:654] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:38.216256 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c96d4c2dd6494cabfdf34090b636dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:38.216262 2676 ts_tablet_manager.cc:1939] T e45e3c7e24604c6d9436cf774667154c P 3ba19504f8764362b7e9e6ea65d8885c: Deleting consensus metadata
I20260620 12:19:38.217435 2110 catalog_manager.cc:4841] TS 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705): tablet e45e3c7e24604c6d9436cf774667154c (table test-table [id=e7f5305227514e06b3d5c331e5dc839f]) successfully deleted
I20260620 12:19:38.217586 2633 tablet_bootstrap.cc:492] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: No bootstrap required, opened a new log
I20260620 12:19:38.217655 2633 ts_tablet_manager.cc:1397] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:38.218155 2633 raft_consensus.cc:348] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.218257 2633 raft_consensus.cc:374] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:38.218279 2631 tablet_bootstrap.cc:492] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: No bootstrap required, opened a new log
I20260620 12:19:38.218291 2633 raft_consensus.cc:729] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba19504f8764362b7e9e6ea65d8885c, State: Initialized, Role: FOLLOWER
I20260620 12:19:38.218348 2632 tablet_bootstrap.cc:492] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: Bootstrap starting.
I20260620 12:19:38.218366 2631 ts_tablet_manager.cc:1397] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 12:19:38.218415 2633 consensus_queue.cc:260] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.218657 2633 ts_tablet_manager.cc:1428] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:38.218783 2631 raft_consensus.cc:348] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.218897 2631 raft_consensus.cc:374] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:38.218927 2631 raft_consensus.cc:729] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1dd6cae7d7422e999b1b33cda56666, State: Initialized, Role: FOLLOWER
I20260620 12:19:38.219029 2631 consensus_queue.cc:260] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.219123 2632 tablet_bootstrap.cc:654] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:38.219280 2631 ts_tablet_manager.cc:1428] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:38.220320 2632 tablet_bootstrap.cc:492] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: No bootstrap required, opened a new log
I20260620 12:19:38.220403 2632 ts_tablet_manager.cc:1397] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 12:19:38.220758 2632 raft_consensus.cc:348] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.220880 2632 raft_consensus.cc:374] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:38.220921 2632 raft_consensus.cc:729] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Initialized, Role: FOLLOWER
I20260620 12:19:38.221065 2632 consensus_queue.cc:260] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.221311 2632 ts_tablet_manager.cc:1428] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:38.409147 2637 raft_consensus.cc:482] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:38.409303 2637 raft_consensus.cc:504] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.410323 2637 leader_election.cc:288] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2:45725), 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:38.415669 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.410675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51359 (local address 127.0.98.2:45725)
0620 12:19:38.410813 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:38.410818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:38.411202 (+ 384us) server_negotiation.cc:407] Connection header received
0620 12:19:38.411267 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:38.411270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:38.411322 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:38.411419 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:38.411682 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.412487 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.414291 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.414916 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.415029 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.415094 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:38.415464 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 12:19:38.415535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:19:38.415679 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.411043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43537 (local address 127.0.98.3:39705)
0620 12:19:38.411345 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:38.411355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:38.411370 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:19:38.411420 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:38.411422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:38.411462 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:38.411543 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:38.411796 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.412552 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.414306 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.414943 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.415035 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.415093 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:38.415464 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 12:19:38.415535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260620 12:19:38.415704 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.410932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39705 (local address 127.0.98.1:43537)
0620 12:19:38.411194 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:38.411208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:38.411296 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:38.411564 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:38.411567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:38.411580 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:38.411691 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.411698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.412663 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.412665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.414155 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.414166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.415072 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.415076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.415291 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.415354 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 12:19:38.415433 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:19:38.415528 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":151,"thread_start_us":125,"threads_started":1}
I20260620 12:19:38.415711 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.410537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45725 (local address 127.0.98.1:51359)
0620 12:19:38.410809 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:38.410826 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:38.410934 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:38.411422 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:38.411424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:38.411435 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:38.411564 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.411570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.412587 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.412590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.414155 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.414166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.415072 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.415076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.415316 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.415355 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 12:19:38.415433 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 12:19:38.415528 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":198,"thread_start_us":94,"threads_started":1}
I20260620 12:19:38.416291 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c96d4c2dd6494cabfdf34090b636dc" candidate_uuid: "4c1dd6cae7d7422e999b1b33cda56666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" is_pre_election: true
I20260620 12:19:38.416353 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c96d4c2dd6494cabfdf34090b636dc" candidate_uuid: "4c1dd6cae7d7422e999b1b33cda56666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c" is_pre_election: true
I20260620 12:19:38.416492 2550 raft_consensus.cc:2454] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1dd6cae7d7422e999b1b33cda56666 in term 0.
I20260620 12:19:38.416536 2422 raft_consensus.cc:2454] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1dd6cae7d7422e999b1b33cda56666 in term 0.
I20260620 12:19:38.416860 2214 leader_election.cc:302] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba19504f8764362b7e9e6ea65d8885c, 4c1dd6cae7d7422e999b1b33cda56666; no voters:
I20260620 12:19:38.417110 2637 raft_consensus.cc:2781] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:38.417174 2637 raft_consensus.cc:482] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:38.417201 2637 raft_consensus.cc:3037] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:38.418154 2637 raft_consensus.cc:504] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.418483 2637 leader_election.cc:288] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [CANDIDATE]: Term 1 election: Requested vote from peers 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2:45725), 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:38.418905 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c96d4c2dd6494cabfdf34090b636dc" candidate_uuid: "4c1dd6cae7d7422e999b1b33cda56666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa347d18f3a4824b04ab1f58e3d7952"
I20260620 12:19:38.418932 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c96d4c2dd6494cabfdf34090b636dc" candidate_uuid: "4c1dd6cae7d7422e999b1b33cda56666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c"
I20260620 12:19:38.419030 2422 raft_consensus.cc:3037] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:38.419044 2550 raft_consensus.cc:3037] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:38.419911 2422 raft_consensus.cc:2454] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1dd6cae7d7422e999b1b33cda56666 in term 1.
I20260620 12:19:38.419911 2550 raft_consensus.cc:2454] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1dd6cae7d7422e999b1b33cda56666 in term 1.
I20260620 12:19:38.420190 2214 leader_election.cc:302] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1dd6cae7d7422e999b1b33cda56666, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:38.420361 2637 raft_consensus.cc:2781] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:38.420578 2637 raft_consensus.cc:686] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1dd6cae7d7422e999b1b33cda56666, State: Running, Role: LEADER
I20260620 12:19:38.420835 2637 consensus_queue.cc:237] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:38.423074 2125 catalog_manager.cc:5495] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4c1dd6cae7d7422e999b1b33cda56666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } health_report { overall_health: UNKNOWN } } }
I20260620 12:19:38.470214 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.465968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.0.98.1:34143)
0620 12:19:38.466220 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:38.466226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:38.466238 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:19:38.466297 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:38.466300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:38.466346 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:38.466444 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:38.466717 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.467304 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.468480 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.468708 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.468795 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.469111 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.469549 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.469640 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:38.469747 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 12:19:38.469791 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260620 12:19:38.480679 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.477019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45725 (local address 127.0.98.1:33831)
0620 12:19:38.477367 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:38.477382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:38.477478 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:38.477838 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:38.477841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:38.477848 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:19:38.477959 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.477964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.478647 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.478650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.479634 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.479643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.480068 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.480071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.480142 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.480180 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.480444 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.480449 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:19:38.480503 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 12:19:38.480567 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20260620 12:19:38.480824 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.477200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33831 (local address 127.0.98.2:45725)
0620 12:19:38.477335 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:38.477339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:38.477468 (+ 129us) server_negotiation.cc:407] Connection header received
0620 12:19:38.477666 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:38.477669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:38.477726 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:38.477813 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:38.478079 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.478551 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.479748 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.479965 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.480044 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.480291 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.480374 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.480426 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:38.480656 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 12:19:38.480700 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:19:38.486050 2550 raft_consensus.cc:1261] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Refusing update from remote peer 4c1dd6cae7d7422e999b1b33cda56666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:38.486325 2422 raft_consensus.cc:1261] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Refusing update from remote peer 4c1dd6cae7d7422e999b1b33cda56666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:38.486514 2637 consensus_queue.cc:1035] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:38.486799 2684 consensus_queue.cc:1035] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:38.503762 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.500684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52039 (local address 127.0.98.1:34143)
0620 12:19:38.500787 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:38.500791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:38.500806 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:19:38.500913 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:38.500918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:38.500968 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:38.501067 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:38.501342 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.501831 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.502690 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.502994 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.503105 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.503345 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.503416 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.503494 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:38.503591 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 12:19:38.503636 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 12:19:38.503765 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:38.500623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34143 (local address 127.0.98.2:52039)
0620 12:19:38.500748 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:38.500763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:38.500837 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:38.501090 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:38.501093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:38.501100 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:19:38.501214 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.501222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.501934 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.501940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.502570 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:38.502579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:38.503102 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:38.503105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:38.503191 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:38.503227 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:19:38.503508 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:19:38.503516 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:19:38.503573 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 12:19:38.503655 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 12:19:38.504401 2660 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 12:19:40.063184 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:40.063335 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:40.065495 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: Sending DeleteTablet for 3 replicas of tablet d0c96d4c2dd6494cabfdf34090b636dc
I20260620 12:19:40.066054 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c96d4c2dd6494cabfdf34090b636dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47530
I20260620 12:19:40.066045 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c96d4c2dd6494cabfdf34090b636dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38286
I20260620 12:19:40.066092 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c96d4c2dd6494cabfdf34090b636dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41254
I20260620 12:19:40.066386 2718 tablet_replica.cc:330] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: stopping tablet replica
I20260620 12:19:40.066404 2716 tablet_replica.cc:330] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: stopping tablet replica
I20260620 12:19:40.066522 2718 raft_consensus.cc:2229] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:40.066574 2716 raft_consensus.cc:2229] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:40.066634 2718 raft_consensus.cc:2258] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:40.066715 2716 raft_consensus.cc:2258] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:40.067404 2718 ts_tablet_manager.cc:1905] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:40.067432 2716 ts_tablet_manager.cc:1905] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:40.070708 2718 ts_tablet_manager.cc:1918] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:19:40.070799 2718 log.cc:1196] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/d0c96d4c2dd6494cabfdf34090b636dc
I20260620 12:19:40.071072 2718 ts_tablet_manager.cc:1939] T d0c96d4c2dd6494cabfdf34090b636dc P 3ba19504f8764362b7e9e6ea65d8885c: Deleting consensus metadata
I20260620 12:19:40.071338 2716 ts_tablet_manager.cc:1918] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:19:40.071424 2716 log.cc:1196] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/d0c96d4c2dd6494cabfdf34090b636dc
I20260620 12:19:40.071713 2716 ts_tablet_manager.cc:1939] T d0c96d4c2dd6494cabfdf34090b636dc P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting consensus metadata
I20260620 12:19:40.071904 2110 catalog_manager.cc:4841] TS 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705): tablet d0c96d4c2dd6494cabfdf34090b636dc (table test-table [id=0455224cbfd740e5a12e9ec241dfc5ae]) successfully deleted
I20260620 12:19:40.072345 2717 tablet_replica.cc:330] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: stopping tablet replica
I20260620 12:19:40.072467 2717 raft_consensus.cc:2229] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:19:40.072570 2110 catalog_manager.cc:4841] TS 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2:45725): tablet d0c96d4c2dd6494cabfdf34090b636dc (table test-table [id=0455224cbfd740e5a12e9ec241dfc5ae]) successfully deleted
I20260620 12:19:40.072813 2717 raft_consensus.cc:2258] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:40.073521 2717 ts_tablet_manager.cc:1905] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:40.075572 2717 ts_tablet_manager.cc:1918] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:19:40.075636 2717 log.cc:1196] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/d0c96d4c2dd6494cabfdf34090b636dc
I20260620 12:19:40.075852 2717 ts_tablet_manager.cc:1939] T d0c96d4c2dd6494cabfdf34090b636dc P 4c1dd6cae7d7422e999b1b33cda56666: Deleting consensus metadata
I20260620 12:19:40.076514 2110 catalog_manager.cc:4841] TS 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143): tablet d0c96d4c2dd6494cabfdf34090b636dc (table test-table [id=0455224cbfd740e5a12e9ec241dfc5ae]) successfully deleted
Jun 20 12:19:40 dist-test-slave-9b3m krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957980, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:19:40 dist-test-slave-9b3m krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957980, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:40.098191 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:40.085391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.0.98.62:39331)
0620 12:19:40.085666 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:40.085673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:40.085687 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:19:40.085778 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:40.085780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:40.085820 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:40.085893 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:40.086628 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:40.087220 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:40.088176 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:40.088409 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:40.088497 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:40.095215 (+ 6718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:40.095218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:40.095220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:40.095242 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:40.096605 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:40.096978 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:40.096980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:40.096981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:40.097155 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:40.097412 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:40.097414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:40.097415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:40.097583 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:40.097680 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:40.097880 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 12:19:40.097975 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
I20260620 12:19:40.100589 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:19:40.100958 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:40.105345 2262 tablet_service.cc:1467] Processing CreateTablet for tablet d143b4e29e144c57aa99142959b12642 (DEFAULT_TABLE table=test-table [id=09ae1c8a6cdd47fc9b78539fd1248f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:40.105433 2542 tablet_service.cc:1467] Processing CreateTablet for tablet d143b4e29e144c57aa99142959b12642 (DEFAULT_TABLE table=test-table [id=09ae1c8a6cdd47fc9b78539fd1248f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:40.105396 2401 tablet_service.cc:1467] Processing CreateTablet for tablet d143b4e29e144c57aa99142959b12642 (DEFAULT_TABLE table=test-table [id=09ae1c8a6cdd47fc9b78539fd1248f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:40.105639 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d143b4e29e144c57aa99142959b12642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:40.105672 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d143b4e29e144c57aa99142959b12642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:40.105734 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d143b4e29e144c57aa99142959b12642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:40.107724 2727 tablet_bootstrap.cc:492] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: Bootstrap starting.
I20260620 12:19:40.108098 2729 tablet_bootstrap.cc:492] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: Bootstrap starting.
I20260620 12:19:40.108649 2727 tablet_bootstrap.cc:654] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:40.109249 2728 tablet_bootstrap.cc:492] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: Bootstrap starting.
I20260620 12:19:40.109424 2729 tablet_bootstrap.cc:654] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:40.109789 2727 tablet_bootstrap.cc:492] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: No bootstrap required, opened a new log
I20260620 12:19:40.109877 2727 ts_tablet_manager.cc:1397] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 12:19:40.110140 2728 tablet_bootstrap.cc:654] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:40.110320 2727 raft_consensus.cc:348] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.110438 2727 raft_consensus.cc:374] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:40.110471 2727 raft_consensus.cc:729] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Initialized, Role: FOLLOWER
I20260620 12:19:40.110625 2729 tablet_bootstrap.cc:492] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: No bootstrap required, opened a new log
I20260620 12:19:40.110632 2727 consensus_queue.cc:260] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.110706 2729 ts_tablet_manager.cc:1397] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:19:40.110926 2727 ts_tablet_manager.cc:1428] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:19:40.111140 2729 raft_consensus.cc:348] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.111268 2729 raft_consensus.cc:374] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:40.111303 2729 raft_consensus.cc:729] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba19504f8764362b7e9e6ea65d8885c, State: Initialized, Role: FOLLOWER
I20260620 12:19:40.111455 2728 tablet_bootstrap.cc:492] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: No bootstrap required, opened a new log
I20260620 12:19:40.111455 2729 consensus_queue.cc:260] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.111547 2728 ts_tablet_manager.cc:1397] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:40.111833 2729 ts_tablet_manager.cc:1428] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:40.111945 2728 raft_consensus.cc:348] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.112047 2728 raft_consensus.cc:374] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:40.112097 2728 raft_consensus.cc:729] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1dd6cae7d7422e999b1b33cda56666, State: Initialized, Role: FOLLOWER
I20260620 12:19:40.112226 2728 consensus_queue.cc:260] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.112507 2728 ts_tablet_manager.cc:1428] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:40.209540 2703 raft_consensus.cc:482] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:40.209657 2703 raft_consensus.cc:504] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.210037 2703 leader_election.cc:288] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143), 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:40.210424 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d143b4e29e144c57aa99142959b12642" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1dd6cae7d7422e999b1b33cda56666" is_pre_election: true
I20260620 12:19:40.210510 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d143b4e29e144c57aa99142959b12642" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c" is_pre_election: true
I20260620 12:19:40.210598 2282 raft_consensus.cc:2454] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 0.
I20260620 12:19:40.210705 2550 raft_consensus.cc:2454] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 0.
I20260620 12:19:40.210935 2354 leader_election.cc:302] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1dd6cae7d7422e999b1b33cda56666, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:40.211131 2703 raft_consensus.cc:2781] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:40.211194 2703 raft_consensus.cc:482] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:40.211232 2703 raft_consensus.cc:3037] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:40.212206 2703 raft_consensus.cc:504] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.212512 2703 leader_election.cc:288] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143), 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:40.212888 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d143b4e29e144c57aa99142959b12642" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c"
I20260620 12:19:40.212879 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d143b4e29e144c57aa99142959b12642" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1dd6cae7d7422e999b1b33cda56666"
I20260620 12:19:40.213020 2550 raft_consensus.cc:3037] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:40.213073 2282 raft_consensus.cc:3037] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:40.213897 2550 raft_consensus.cc:2454] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 1.
I20260620 12:19:40.214201 2282 raft_consensus.cc:2454] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 1.
I20260620 12:19:40.214226 2354 leader_election.cc:302] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba19504f8764362b7e9e6ea65d8885c, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:40.214478 2703 raft_consensus.cc:2781] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:40.214669 2703 raft_consensus.cc:686] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Running, Role: LEADER
I20260620 12:19:40.214861 2703 consensus_queue.cc:237] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.216693 2122 catalog_manager.cc:5495] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } health_report { overall_health: UNKNOWN } } }
I20260620 12:19:40.239711 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:40.236436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.98.2:45725)
0620 12:19:40.236706 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:40.236713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:40.236725 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:19:40.236762 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:40.236763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:40.236801 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:40.236872 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:40.237296 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:40.237836 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:40.238688 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:40.238911 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:40.239002 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:40.239259 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:40.239335 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:40.239406 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:40.239506 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 12:19:40.239593 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
I20260620 12:19:40.245677 2550 raft_consensus.cc:1261] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Refusing update from remote peer 8fa347d18f3a4824b04ab1f58e3d7952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:40.245705 2282 raft_consensus.cc:1261] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Refusing update from remote peer 8fa347d18f3a4824b04ab1f58e3d7952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:40.246064 2694 consensus_queue.cc:1035] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:40.246237 2697 consensus_queue.cc:1035] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:40.282847 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:40.282980 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:40.284824 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: Sending DeleteTablet for 3 replicas of tablet d143b4e29e144c57aa99142959b12642
I20260620 12:19:40.285307 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet d143b4e29e144c57aa99142959b12642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47530
I20260620 12:19:40.285399 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet d143b4e29e144c57aa99142959b12642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41254
I20260620 12:19:40.285425 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet d143b4e29e144c57aa99142959b12642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38286
I20260620 12:19:40.285471 2717 tablet_replica.cc:330] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: stopping tablet replica
I20260620 12:19:40.285596 2717 raft_consensus.cc:2229] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:40.285647 2716 tablet_replica.cc:330] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: stopping tablet replica
I20260620 12:19:40.285712 2717 raft_consensus.cc:2258] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:40.285775 2716 raft_consensus.cc:2229] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:19:40.285883 2718 tablet_replica.cc:330] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: stopping tablet replica
I20260620 12:19:40.285934 2716 raft_consensus.cc:2258] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:40.285985 2718 raft_consensus.cc:2229] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:40.285953 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:19:40.286084 2718 raft_consensus.cc:2258] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 12:19:40.286343 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:40.286689 2718 ts_tablet_manager.cc:1905] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:40.286760 2716 ts_tablet_manager.cc:1905] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:40.286789 2717 ts_tablet_manager.cc:1905] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:40.289012 2716 ts_tablet_manager.cc:1918] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:40.289129 2716 log.cc:1196] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/d143b4e29e144c57aa99142959b12642
I20260620 12:19:40.289114 2718 ts_tablet_manager.cc:1918] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:40.289201 2718 log.cc:1196] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/d143b4e29e144c57aa99142959b12642
I20260620 12:19:40.289198 2717 ts_tablet_manager.cc:1918] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:19:40.289258 2717 log.cc:1196] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/d143b4e29e144c57aa99142959b12642
I20260620 12:19:40.289333 2716 ts_tablet_manager.cc:1939] T d143b4e29e144c57aa99142959b12642 P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting consensus metadata
I20260620 12:19:40.289471 2718 ts_tablet_manager.cc:1939] T d143b4e29e144c57aa99142959b12642 P 3ba19504f8764362b7e9e6ea65d8885c: Deleting consensus metadata
I20260620 12:19:40.289536 2717 ts_tablet_manager.cc:1939] T d143b4e29e144c57aa99142959b12642 P 4c1dd6cae7d7422e999b1b33cda56666: Deleting consensus metadata
I20260620 12:19:40.289835 2110 catalog_manager.cc:4841] TS 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2:45725): tablet d143b4e29e144c57aa99142959b12642 (table test-table [id=09ae1c8a6cdd47fc9b78539fd1248f53]) successfully deleted
I20260620 12:19:40.290119 2110 catalog_manager.cc:4841] TS 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143): tablet d143b4e29e144c57aa99142959b12642 (table test-table [id=09ae1c8a6cdd47fc9b78539fd1248f53]) successfully deleted
I20260620 12:19:40.290509 2110 catalog_manager.cc:4841] TS 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705): tablet d143b4e29e144c57aa99142959b12642 (table test-table [id=09ae1c8a6cdd47fc9b78539fd1248f53]) successfully deleted
I20260620 12:19:40.293311 2262 tablet_service.cc:1467] Processing CreateTablet for tablet da40011f8ad34b3d8ecc5faacb0037ce (DEFAULT_TABLE table=test-table [id=48d7c0cac4d6432b81487cc4cc80937d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:40.293383 2401 tablet_service.cc:1467] Processing CreateTablet for tablet da40011f8ad34b3d8ecc5faacb0037ce (DEFAULT_TABLE table=test-table [id=48d7c0cac4d6432b81487cc4cc80937d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:40.293591 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da40011f8ad34b3d8ecc5faacb0037ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:40.293637 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da40011f8ad34b3d8ecc5faacb0037ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:40.295239 2728 tablet_bootstrap.cc:492] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666: Bootstrap starting.
I20260620 12:19:40.295356 2727 tablet_bootstrap.cc:492] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: Bootstrap starting.
I20260620 12:19:40.295501 2542 tablet_service.cc:1467] Processing CreateTablet for tablet da40011f8ad34b3d8ecc5faacb0037ce (DEFAULT_TABLE table=test-table [id=48d7c0cac4d6432b81487cc4cc80937d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:40.295738 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da40011f8ad34b3d8ecc5faacb0037ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:40.296030 2727 tablet_bootstrap.cc:654] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:40.296030 2728 tablet_bootstrap.cc:654] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:40.297101 2727 tablet_bootstrap.cc:492] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: No bootstrap required, opened a new log
I20260620 12:19:40.297188 2727 ts_tablet_manager.cc:1397] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 12:19:40.297394 2728 tablet_bootstrap.cc:492] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666: No bootstrap required, opened a new log
I20260620 12:19:40.297477 2728 ts_tablet_manager.cc:1397] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:40.297587 2727 raft_consensus.cc:348] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.297703 2727 raft_consensus.cc:374] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:40.297747 2727 raft_consensus.cc:729] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Initialized, Role: FOLLOWER
I20260620 12:19:40.297868 2729 tablet_bootstrap.cc:492] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: Bootstrap starting.
I20260620 12:19:40.297859 2727 consensus_queue.cc:260] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.297891 2728 raft_consensus.cc:348] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.297981 2728 raft_consensus.cc:374] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:40.298022 2728 raft_consensus.cc:729] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1dd6cae7d7422e999b1b33cda56666, State: Initialized, Role: FOLLOWER
I20260620 12:19:40.298126 2727 ts_tablet_manager.cc:1428] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:40.298138 2728 consensus_queue.cc:260] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.298401 2728 ts_tablet_manager.cc:1428] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:19:40.298779 2729 tablet_bootstrap.cc:654] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:40.299911 2729 tablet_bootstrap.cc:492] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: No bootstrap required, opened a new log
I20260620 12:19:40.299989 2729 ts_tablet_manager.cc:1397] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:40.300387 2729 raft_consensus.cc:348] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.300495 2729 raft_consensus.cc:374] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:40.300534 2729 raft_consensus.cc:729] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba19504f8764362b7e9e6ea65d8885c, State: Initialized, Role: FOLLOWER
I20260620 12:19:40.300664 2729 consensus_queue.cc:260] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.300945 2729 ts_tablet_manager.cc:1428] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:19:40.436307 2697 raft_consensus.cc:482] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:40.436446 2697 raft_consensus.cc:504] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.436816 2697 leader_election.cc:288] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143), 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:40.437373 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da40011f8ad34b3d8ecc5faacb0037ce" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1dd6cae7d7422e999b1b33cda56666" is_pre_election: true
I20260620 12:19:40.437429 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da40011f8ad34b3d8ecc5faacb0037ce" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c" is_pre_election: true
I20260620 12:19:40.437508 2282 raft_consensus.cc:2454] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 0.
I20260620 12:19:40.437644 2550 raft_consensus.cc:2454] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 0.
I20260620 12:19:40.437889 2354 leader_election.cc:302] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1dd6cae7d7422e999b1b33cda56666, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:40.438076 2697 raft_consensus.cc:2781] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:40.438138 2697 raft_consensus.cc:482] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:40.438192 2697 raft_consensus.cc:3037] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:40.439177 2697 raft_consensus.cc:504] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.439494 2697 leader_election.cc:288] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143), 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705)
I20260620 12:19:40.439874 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da40011f8ad34b3d8ecc5faacb0037ce" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1dd6cae7d7422e999b1b33cda56666"
I20260620 12:19:40.439873 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da40011f8ad34b3d8ecc5faacb0037ce" candidate_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba19504f8764362b7e9e6ea65d8885c"
I20260620 12:19:40.439997 2550 raft_consensus.cc:3037] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:40.439997 2282 raft_consensus.cc:3037] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:40.441192 2282 raft_consensus.cc:2454] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 1.
I20260620 12:19:40.441192 2550 raft_consensus.cc:2454] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa347d18f3a4824b04ab1f58e3d7952 in term 1.
I20260620 12:19:40.441499 2354 leader_election.cc:302] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba19504f8764362b7e9e6ea65d8885c, 8fa347d18f3a4824b04ab1f58e3d7952; no voters:
I20260620 12:19:40.441669 2697 raft_consensus.cc:2781] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:40.441758 2697 raft_consensus.cc:686] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa347d18f3a4824b04ab1f58e3d7952, State: Running, Role: LEADER
I20260620 12:19:40.441900 2697 consensus_queue.cc:237] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } }
I20260620 12:19:40.443529 2125 catalog_manager.cc:5495] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fa347d18f3a4824b04ab1f58e3d7952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 } health_report { overall_health: UNKNOWN } } }
I20260620 12:19:40.459632 2550 raft_consensus.cc:1261] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Refusing update from remote peer 8fa347d18f3a4824b04ab1f58e3d7952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:40.459681 2282 raft_consensus.cc:1261] T da40011f8ad34b3d8ecc5faacb0037ce P 4c1dd6cae7d7422e999b1b33cda56666 [term 1 FOLLOWER]: Refusing update from remote peer 8fa347d18f3a4824b04ab1f58e3d7952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:40.460016 2694 consensus_queue.cc:1035] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba19504f8764362b7e9e6ea65d8885c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:40.460192 2703 consensus_queue.cc:1035] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1dd6cae7d7422e999b1b33cda56666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 12:19:40.472858 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 12:19:42.021299 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:42.021466 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:42.023743 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c36bc4d33994fd28dd40c6bdf4a577a: Sending DeleteTablet for 3 replicas of tablet da40011f8ad34b3d8ecc5faacb0037ce
I20260620 12:19:42.024279 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet da40011f8ad34b3d8ecc5faacb0037ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47530
I20260620 12:19:42.024443 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet da40011f8ad34b3d8ecc5faacb0037ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41254
I20260620 12:19:42.024576 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet da40011f8ad34b3d8ecc5faacb0037ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38286
I20260620 12:19:42.024642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 2197
I20260620 12:19:42.025720 2767 tablet_replica.cc:330] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: stopping tablet replica
I20260620 12:19:42.025872 2767 raft_consensus.cc:2229] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:42.025988 2767 raft_consensus.cc:2258] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:42.026206 2768 tablet_replica.cc:330] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: stopping tablet replica
I20260620 12:19:42.026393 2768 raft_consensus.cc:2229] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:19:42.026633 2768 raft_consensus.cc:2258] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:42.026907 2767 ts_tablet_manager.cc:1905] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:42.027246 2768 ts_tablet_manager.cc:1905] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:42.029111 2767 ts_tablet_manager.cc:1918] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:19:42.029209 2767 log.cc:1196] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/da40011f8ad34b3d8ecc5faacb0037ce
I20260620 12:19:42.029531 2767 ts_tablet_manager.cc:1939] T da40011f8ad34b3d8ecc5faacb0037ce P 3ba19504f8764362b7e9e6ea65d8885c: Deleting consensus metadata
I20260620 12:19:42.030153 2110 catalog_manager.cc:4841] TS 3ba19504f8764362b7e9e6ea65d8885c (127.0.98.3:39705): tablet da40011f8ad34b3d8ecc5faacb0037ce (table test-table [id=48d7c0cac4d6432b81487cc4cc80937d]) successfully deleted
I20260620 12:19:42.030594 2768 ts_tablet_manager.cc:1918] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 12:19:42.030679 2768 log.cc:1196] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/da40011f8ad34b3d8ecc5faacb0037ce
I20260620 12:19:42.030963 2768 ts_tablet_manager.cc:1939] T da40011f8ad34b3d8ecc5faacb0037ce P 8fa347d18f3a4824b04ab1f58e3d7952: Deleting consensus metadata
I20260620 12:19:42.031634 2110 catalog_manager.cc:4841] TS 8fa347d18f3a4824b04ab1f58e3d7952 (127.0.98.2:45725): tablet da40011f8ad34b3d8ecc5faacb0037ce (table test-table [id=48d7c0cac4d6432b81487cc4cc80937d]) successfully deleted
W20260620 12:19:42.033295 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34143 (error 108)
I20260620 12:19:42.033380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 2337
I20260620 12:19:42.035193 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:42.034711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34143 (local address 127.0.0.1:47538)
0620 12:19:42.034971 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:42.035069 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
W20260620 12:19:42.035344 2110 catalog_manager.cc:4584] TS 4c1dd6cae7d7422e999b1b33cda56666 (127.0.98.1:34143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da40011f8ad34b3d8ecc5faacb0037ce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34143: connect: Connection refused (error 111)
I20260620 12:19:42.040918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 2477
I20260620 12:19:42.048424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 2093
2026-06-20T12:19:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6620 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:42 dist-test-slave-9b3m krb5kdc[2773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:42 dist-test-slave-9b3m krb5kdc[2773](info): set up 2 sockets
Jun 20 12:19:42 dist-test-slave-9b3m krb5kdc[2773](info): commencing operation
krb5kdc: starting...
W20260620 12:19:43.106482 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:43 dist-test-slave-9b3m krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:43.232777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:43.340015 2786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:43.340323 2786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:43.340396 2786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:43.344430 2786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:43.344529 2786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:43.344569 2786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:43.344606 2786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:43.344641 2786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:43.349943 2786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:43.351015 2786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:43.352051 2786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:43.360774 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:43.360860 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:43.360900 2786 server_base.cc:1034] running on GCE node
W20260620 12:19:43.360864 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:43.361626 2786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:43.362715 2786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:43.363921 2786 hybrid_clock.cc:648] HybridClock initialized: now 1781957983363904 us; error 35 us; skew 500 ppm
Jun 20 12:19:43 dist-test-slave-9b3m krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:43.366660 2786 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:43.367496 2786 webserver.cc:460] Webserver started at http://127.0.98.62:42241/ using document root <none> and password file <none>
I20260620 12:19:43.368045 2786 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:43.368135 2786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:43.368376 2786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:43.370404 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "abb98bf0e2f24e518b1fb834062b343d"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "c4b0c48f60b3951cd5c65fea295c7f24"
server_key_iv: "ff7ef7682a03c2d731e7af1ee6aa3ec1"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.370956 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "abb98bf0e2f24e518b1fb834062b343d"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "c4b0c48f60b3951cd5c65fea295c7f24"
server_key_iv: "ff7ef7682a03c2d731e7af1ee6aa3ec1"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.374528 2786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 12:19:43.376946 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:43.378181 2786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:43.378330 2786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "abb98bf0e2f24e518b1fb834062b343d"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "c4b0c48f60b3951cd5c65fea295c7f24"
server_key_iv: "ff7ef7682a03c2d731e7af1ee6aa3ec1"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.378458 2786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:43.400029 2786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:43.402837 2786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:43.403035 2786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:43.410202 2786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38439
I20260620 12:19:43.410228 2853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38439 every 8 connection(s)
I20260620 12:19:43.411109 2786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:43.413779 2854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:43.418505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 2786
I20260620 12:19:43.418591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:43.418846 392 external_mini_cluster.cc:1428] Setting key ee9aeea54a99bf36ffec75c00376550e
I20260620 12:19:43.419091 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d: Bootstrap starting.
I20260620 12:19:43.421708 2854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:43.422566 2854 log.cc:824] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:43.424305 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d: No bootstrap required, opened a new log
Jun 20 12:19:43 dist-test-slave-9b3m krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:43.427987 2854 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb98bf0e2f24e518b1fb834062b343d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38439 } }
I20260620 12:19:43.428215 2854 raft_consensus.cc:374] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:43.428292 2854 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb98bf0e2f24e518b1fb834062b343d, State: Initialized, Role: FOLLOWER
I20260620 12:19:43.428790 2854 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb98bf0e2f24e518b1fb834062b343d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38439 } }
I20260620 12:19:43.428956 2854 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:43.429069 2854 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:43.429170 2854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:43.430079 2854 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb98bf0e2f24e518b1fb834062b343d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38439 } }
I20260620 12:19:43.430415 2854 leader_election.cc:302] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb98bf0e2f24e518b1fb834062b343d; no voters:
I20260620 12:19:43.430733 2854 leader_election.cc:288] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:43.430814 2859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:43.430980 2859 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [term 1 LEADER]: Becoming Leader. State: Replica: abb98bf0e2f24e518b1fb834062b343d, State: Running, Role: LEADER
I20260620 12:19:43.431306 2859 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb98bf0e2f24e518b1fb834062b343d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38439 } }
I20260620 12:19:43.431943 2854 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:43.432425 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb98bf0e2f24e518b1fb834062b343d. Latest consensus state: current_term: 1 leader_uuid: "abb98bf0e2f24e518b1fb834062b343d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb98bf0e2f24e518b1fb834062b343d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38439 } } }
I20260620 12:19:43.432605 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:43.432873 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abb98bf0e2f24e518b1fb834062b343d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb98bf0e2f24e518b1fb834062b343d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38439 } } }
I20260620 12:19:43.433089 2865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:43.433599 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:43.420212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.0.98.62:38439)
0620 12:19:43.420627 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:43.420645 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:43.420685 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:19:43.421517 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:43.421540 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:43.421976 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:43.422317 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:43.423012 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.423993 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.424902 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.425428 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.425600 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:43.428253 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:43.428261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:43.428278 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:43.428315 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:43.430266 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:43.430767 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:43.430772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:43.430778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:43.430922 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:43.431256 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:43.431259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:43.431261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:43.431681 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:43.431885 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:43.432174 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 12:19:43.432476 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260620 12:19:43.433102 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:43.436439 2865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:43.441632 2865 catalog_manager.cc:1347] Generated new cluster ID: a7d8d302bdfc491db7b3ed0859837735
I20260620 12:19:43.441699 2865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:43.454759 2865 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:43.456017 2865 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:43.460480 2865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abb98bf0e2f24e518b1fb834062b343d: Generated new TSK 0
I20260620 12:19:43.461571 2865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:43.497674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38439
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:43.607586 2880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:43.607820 2880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:43.607860 2880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:43.611754 2880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:43.611804 2880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:43.611858 2880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:43.616874 2880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:43.617873 2880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:43.618784 2880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:43.628324 2886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:43.628450 2880 server_base.cc:1034] running on GCE node
W20260620 12:19:43.628338 2885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:43.628371 2888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:43.628912 2880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:43.629591 2880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:43.630777 2880 hybrid_clock.cc:648] HybridClock initialized: now 1781957983630762 us; error 28 us; skew 500 ppm
Jun 20 12:19:43 dist-test-slave-9b3m krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:43.633733 2880 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:43.634544 2880 webserver.cc:460] Webserver started at http://127.0.98.1:32843/ using document root <none> and password file <none>
I20260620 12:19:43.635054 2880 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:43.635109 2880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:43.635267 2880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:43.637269 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ee54cb3bef74583b9981a163e56e998"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "3d70919a201baaeafc31f3046c00c85a"
server_key_iv: "ed16c4d152e4ec66fa5e9a574a89a0fb"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.637768 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ee54cb3bef74583b9981a163e56e998"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "3d70919a201baaeafc31f3046c00c85a"
server_key_iv: "ed16c4d152e4ec66fa5e9a574a89a0fb"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.641249 2880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:19:43.643501 2895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:43.644690 2880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:43.644804 2880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "4ee54cb3bef74583b9981a163e56e998"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "3d70919a201baaeafc31f3046c00c85a"
server_key_iv: "ed16c4d152e4ec66fa5e9a574a89a0fb"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.644904 2880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:43.655018 2880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:43.657749 2880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:43.657938 2880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:43.658485 2880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:43.659258 2880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:43.659313 2880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:43.659389 2880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:43.659422 2880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:43.668318 2880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44501
I20260620 12:19:43.668354 3008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44501 every 8 connection(s)
I20260620 12:19:43.669246 2880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:43.673414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 2880
I20260620 12:19:43.673532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:43.673830 392 external_mini_cluster.cc:1428] Setting key 175abbb00a3180c0d61bd92e462ae270
Jun 20 12:19:43 dist-test-slave-9b3m krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:19:43.684208 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:43.671073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55679 (local address 127.0.98.62:38439)
0620 12:19:43.671232 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:43.671236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:43.672075 (+ 839us) server_negotiation.cc:407] Connection header received
0620 12:19:43.673542 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:43.673547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:43.673606 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:43.673702 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:43.675006 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.675782 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.677256 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.677667 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.677792 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:43.680161 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:43.680166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:43.680171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:43.680206 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:43.682056 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:43.682636 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:43.682642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:43.682644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:43.682734 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:43.683064 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:43.683067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:43.683069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:43.683315 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:43.683436 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:43.683935 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 12:19:43.684068 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:19:43.685019 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:43.671455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38439 (local address 127.0.98.1:55679)
0620 12:19:43.671929 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:43.671971 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:43.673320 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:43.673905 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:43.673916 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:43.674275 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:43.674866 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:43.674884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.675911 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.675914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:43.677131 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:43.677142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.677816 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.677819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:43.677957 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:43.678597 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:43.678614 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:43.679993 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:43.682183 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:43.682187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:43.682219 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:43.682535 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:43.682832 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:43.682834 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:43.682835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:43.682964 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:43.683434 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:43.683438 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:43.683651 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 12:19:43.683845 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 12:19:43.684107 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":130,"threads_started":1}
I20260620 12:19:43.686254 3009 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38439
I20260620 12:19:43.686515 3009 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:43.687057 3009 heartbeater.cc:511] Master 127.0.98.62:38439 requested a full tablet report, sending...
I20260620 12:19:43.688645 2818 ts_manager.cc:194] Registered new tserver with Master: 4ee54cb3bef74583b9981a163e56e998 (127.0.98.1:44501)
I20260620 12:19:43.689813 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:43.704077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38439
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:43.810690 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:43.810968 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:43.811044 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:43.815150 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:43.815237 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:43.815328 3014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:43.820967 3014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:43.822037 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:43.823096 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:43.832662 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:43.832647 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:43.832631 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:43.833159 3014 server_base.cc:1034] running on GCE node
I20260620 12:19:43.833591 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:43.834379 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:43.835592 3014 hybrid_clock.cc:648] HybridClock initialized: now 1781957983835577 us; error 27 us; skew 500 ppm
Jun 20 12:19:43 dist-test-slave-9b3m krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:43.838524 3014 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:43.839337 3014 webserver.cc:460] Webserver started at http://127.0.98.2:42891/ using document root <none> and password file <none>
I20260620 12:19:43.839892 3014 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:43.839951 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:43.840116 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:43.842526 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2b282f393f8473fb192e94a174d3d57"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "9d56e7277aaeafd94ae12da091b69818"
server_key_iv: "449cc46affed6ded6193df93045eee24"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.843024 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2b282f393f8473fb192e94a174d3d57"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "9d56e7277aaeafd94ae12da091b69818"
server_key_iv: "449cc46affed6ded6193df93045eee24"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.846644 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:43.849375 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:43.850668 3014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:19:43.850838 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "d2b282f393f8473fb192e94a174d3d57"
format_stamp: "Formatted at 2026-06-20 12:19:43 on dist-test-slave-9b3m"
server_key: "9d56e7277aaeafd94ae12da091b69818"
server_key_iv: "449cc46affed6ded6193df93045eee24"
server_key_version: "encryptionkey@0"
I20260620 12:19:43.850937 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:43.869216 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:43.872790 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:43.873085 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:43.873737 3014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:43.874615 3014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:43.874665 3014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:43.874737 3014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:43.874781 3014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:43.884603 3014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38859
I20260620 12:19:43.884619 3142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38859 every 8 connection(s)
I20260620 12:19:43.885623 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:43.889981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 3014
I20260620 12:19:43.890129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:43.890486 392 external_mini_cluster.cc:1428] Setting key b77ccd0d508485f360cb078abb9cb232
Jun 20 12:19:43 dist-test-slave-9b3m krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:19:43.900310 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:43.887550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45075 (local address 127.0.98.62:38439)
0620 12:19:43.887717 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:43.887741 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:43.889117 (+ 1376us) server_negotiation.cc:407] Connection header received
0620 12:19:43.890238 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:43.890243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:43.890303 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:43.890376 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:43.891961 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.892477 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.893701 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.893922 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.894033 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:43.896795 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:43.896798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:43.896800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:43.896821 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:43.898286 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:43.898786 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:43.898791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:43.898793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:43.898878 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:43.899241 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:43.899244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:43.899246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:43.899429 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:43.899534 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:43.900025 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 12:19:43.900140 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 12:19:43.901255 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:43.887979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38439 (local address 127.0.98.2:45075)
0620 12:19:43.888931 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:43.888973 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:43.889967 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:43.890787 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:43.890796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:43.891267 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:43.891820 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:43.891847 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.892612 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.892615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:43.893572 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:43.893581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:43.894073 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:43.894077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:43.894250 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:43.894896 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:43.894912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:43.896618 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:43.898408 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:43.898414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:43.898429 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:43.898682 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:43.898982 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:43.898987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:43.898989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:43.899138 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:43.899531 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:43.899538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:43.899785 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 12:19:43.899971 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 12:19:43.900287 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":599,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:19:43.902601 3143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38439
I20260620 12:19:43.902853 3143 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:43.903340 3143 heartbeater.cc:511] Master 127.0.98.62:38439 requested a full tablet report, sending...
I20260620 12:19:43.904459 2818 ts_manager.cc:194] Registered new tserver with Master: d2b282f393f8473fb192e94a174d3d57 (127.0.98.2:38859)
I20260620 12:19:43.905237 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:19:43.918581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38439
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:44.027850 3148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:44.028088 3148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:44.028158 3148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:44.032263 3148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:44.032351 3148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:44.032446 3148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:44.038893 3148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:44.040268 3148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:44.041628 3148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:44.051714 3154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:44.051752 3153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:44.051770 3156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:44.051967 3148 server_base.cc:1034] running on GCE node
I20260620 12:19:44.052424 3148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:44.053117 3148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:44.054297 3148 hybrid_clock.cc:648] HybridClock initialized: now 1781957984054275 us; error 46 us; skew 500 ppm
Jun 20 12:19:44 dist-test-slave-9b3m krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:44.057121 3148 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:44.057969 3148 webserver.cc:460] Webserver started at http://127.0.98.3:41723/ using document root <none> and password file <none>
I20260620 12:19:44.058517 3148 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:44.058604 3148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:44.058826 3148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:44.060806 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "e70c78c0aa1e42b5b84cdc7188366f90"
format_stamp: "Formatted at 2026-06-20 12:19:44 on dist-test-slave-9b3m"
server_key: "7acf6b6104d62220b4806717b2cb3088"
server_key_iv: "b14d598c4a43db3b898f4a98e8a67cae"
server_key_version: "encryptionkey@0"
I20260620 12:19:44.061388 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e70c78c0aa1e42b5b84cdc7188366f90"
format_stamp: "Formatted at 2026-06-20 12:19:44 on dist-test-slave-9b3m"
server_key: "7acf6b6104d62220b4806717b2cb3088"
server_key_iv: "b14d598c4a43db3b898f4a98e8a67cae"
server_key_version: "encryptionkey@0"
I20260620 12:19:44.064985 3148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:19:44.067325 3163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:44.068491 3148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:19:44.068619 3148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "e70c78c0aa1e42b5b84cdc7188366f90"
format_stamp: "Formatted at 2026-06-20 12:19:44 on dist-test-slave-9b3m"
server_key: "7acf6b6104d62220b4806717b2cb3088"
server_key_iv: "b14d598c4a43db3b898f4a98e8a67cae"
server_key_version: "encryptionkey@0"
I20260620 12:19:44.068723 3148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:44.107299 3148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:44.110260 3148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:44.110482 3148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:44.111080 3148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:44.111896 3148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:44.111964 3148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:44.112036 3148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:44.112080 3148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:44.120780 3148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33161
I20260620 12:19:44.120803 3276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33161 every 8 connection(s)
I20260620 12:19:44.121683 3148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:44.125167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 3148
I20260620 12:19:44.125253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:44.125522 392 external_mini_cluster.cc:1428] Setting key 50e5414b2efc080a9eaa4d3d98e11aa2
Jun 20 12:19:44 dist-test-slave-9b3m krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:44.136879 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:44.123672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55867 (local address 127.0.98.62:38439)
0620 12:19:44.123804 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:44.123808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:44.124645 (+ 837us) server_negotiation.cc:407] Connection header received
0620 12:19:44.126231 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:44.126234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:44.126303 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:44.126390 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:44.127903 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:44.128445 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:44.130107 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:44.130318 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:44.130412 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:44.132728 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:44.132732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:44.132735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:44.132785 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:44.134176 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:44.134926 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:44.134933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:44.134935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:44.135040 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:44.135465 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:44.135470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:44.135473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:44.135788 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:44.135911 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:44.136552 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 12:19:44.136710 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:19:44.137652 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:44.124058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38439 (local address 127.0.98.3:55867)
0620 12:19:44.124503 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:44.124545 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:44.126013 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:44.126586 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:44.126593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:44.127088 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:44.127750 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:44.127769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:44.128590 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:44.128593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:44.129998 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:44.130007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:44.130448 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:44.130451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:44.130581 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:44.131091 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:44.131107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:44.132562 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:44.134353 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:44.134359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:44.134389 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:44.134803 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:44.135145 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:44.135148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:44.135150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:44.135346 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:44.135907 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:44.135913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:44.136251 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 12:19:44.136475 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 12:19:44.136805 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260620 12:19:44.138886 3277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38439
I20260620 12:19:44.139145 3277 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:44.139655 3277 heartbeater.cc:511] Master 127.0.98.62:38439 requested a full tablet report, sending...
I20260620 12:19:44.140633 2817 ts_manager.cc:194] Registered new tserver with Master: e70c78c0aa1e42b5b84cdc7188366f90 (127.0.98.3:33161)
I20260620 12:19:44.141261 2817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55867
I20260620 12:19:44.150969 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:19:44.163623 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:19:44.162186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.0.98.62:38439)
0620 12:19:44.162324 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:44.162329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:44.162484 (+ 155us) server_negotiation.cc:407] Connection header received
0620 12:19:44.162652 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:44.162654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:44.162695 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:44.162787 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:44.163480 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52672: BlockingRecv error: recv got EOF from 127.0.0.1:52672 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:19:44.164875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 2880
I20260620 12:19:44.170280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 3014
I20260620 12:19:44.175549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 3148
I20260620 12:19:44.180954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 2786
2026-06-20T12:19:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2131 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:44 dist-test-slave-9b3m krb5kdc[3289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:44 dist-test-slave-9b3m krb5kdc[3289](info): set up 2 sockets
Jun 20 12:19:44 dist-test-slave-9b3m krb5kdc[3289](info): commencing operation
krb5kdc: starting...
W20260620 12:19:45.228935 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:45 dist-test-slave-9b3m krb5kdc[3289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:45.352034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35783
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:45.460395 3302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:45.460731 3302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:45.460819 3302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:45.464843 3302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:45.464939 3302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:45.464978 3302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:45.465016 3302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:45.465065 3302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:45.470856 3302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:45.471909 3302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:45.472945 3302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:19:45.481276 3302 server_base.cc:1034] running on GCE node
W20260620 12:19:45.481293 3307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:45.481318 3308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:45.481432 3310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:45.481945 3302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:45.483032 3302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:45.484241 3302 hybrid_clock.cc:648] HybridClock initialized: now 1781957985484193 us; error 43 us; skew 500 ppm
Jun 20 12:19:45 dist-test-slave-9b3m krb5kdc[3289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:45.490636 3302 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:45.491606 3302 webserver.cc:460] Webserver started at http://127.0.98.62:36725/ using document root <none> and password file <none>
I20260620 12:19:45.492244 3302 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:45.492347 3302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:45.492594 3302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:45.494876 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "26461cbf4c204adbb639183a4a1d29ba"
format_stamp: "Formatted at 2026-06-20 12:19:45 on dist-test-slave-9b3m"
server_key: "81b60d0c556fdd2ee2c08b8172c6d3b4"
server_key_iv: "a0320c5a0566fcbaaaa1d46fcec383cd"
server_key_version: "encryptionkey@0"
I20260620 12:19:45.495472 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "26461cbf4c204adbb639183a4a1d29ba"
format_stamp: "Formatted at 2026-06-20 12:19:45 on dist-test-slave-9b3m"
server_key: "81b60d0c556fdd2ee2c08b8172c6d3b4"
server_key_iv: "a0320c5a0566fcbaaaa1d46fcec383cd"
server_key_version: "encryptionkey@0"
I20260620 12:19:45.499466 3302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 12:19:45.502626 3317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:45.503976 3302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:45.504151 3302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "26461cbf4c204adbb639183a4a1d29ba"
format_stamp: "Formatted at 2026-06-20 12:19:45 on dist-test-slave-9b3m"
server_key: "81b60d0c556fdd2ee2c08b8172c6d3b4"
server_key_iv: "a0320c5a0566fcbaaaa1d46fcec383cd"
server_key_version: "encryptionkey@0"
I20260620 12:19:45.504271 3302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:45.524160 3302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:45.527566 3302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:45.527818 3302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:45.535040 3302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35783
I20260620 12:19:45.535359 3369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35783 every 8 connection(s)
I20260620 12:19:45.536015 3302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:45.538101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 3302
I20260620 12:19:45.538197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:45.538465 392 external_mini_cluster.cc:1428] Setting key ab9c27267f45f704c8eaa1ab58ecf99e
I20260620 12:19:45.538996 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:45.546128 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba: Bootstrap starting.
Jun 20 12:19:45 dist-test-slave-9b3m krb5kdc[3289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:45.548583 3370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:45.549526 3370 log.cc:824] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:45.551753 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba: No bootstrap required, opened a new log
I20260620 12:19:45.554312 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:45.539846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.0.98.62:35783)
0620 12:19:45.540274 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:45.540294 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:45.540337 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:19:45.541237 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:45.541261 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:45.541693 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:45.542092 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:45.542747 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:45.544241 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:45.545208 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:45.545923 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:45.546084 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:45.548906 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:45.548912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:45.548928 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:45.548959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:45.550843 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:45.551324 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:45.551330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:45.551336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:45.551476 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:45.551825 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:45.551830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:45.551831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:45.552343 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:45.552529 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:45.552941 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 12:19:45.553266 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260620 12:19:45.556311 3370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26461cbf4c204adbb639183a4a1d29ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35783 } }
I20260620 12:19:45.556584 3370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:45.556674 3370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26461cbf4c204adbb639183a4a1d29ba, State: Initialized, Role: FOLLOWER
I20260620 12:19:45.557206 3370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26461cbf4c204adbb639183a4a1d29ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35783 } }
I20260620 12:19:45.557358 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:19:45.557400 3370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:45.557469 3370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:45.557564 3370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:45.558521 3370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26461cbf4c204adbb639183a4a1d29ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35783 } }
I20260620 12:19:45.558881 3370 leader_election.cc:302] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26461cbf4c204adbb639183a4a1d29ba; no voters:
I20260620 12:19:45.559234 3370 leader_election.cc:288] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:45.559314 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:45.559521 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [term 1 LEADER]: Becoming Leader. State: Replica: 26461cbf4c204adbb639183a4a1d29ba, State: Running, Role: LEADER
I20260620 12:19:45.560194 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26461cbf4c204adbb639183a4a1d29ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35783 } }
I20260620 12:19:45.560604 3370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:45.561420 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26461cbf4c204adbb639183a4a1d29ba. Latest consensus state: current_term: 1 leader_uuid: "26461cbf4c204adbb639183a4a1d29ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26461cbf4c204adbb639183a4a1d29ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35783 } } }
I20260620 12:19:45.561546 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:45.561914 3385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:45.564211 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26461cbf4c204adbb639183a4a1d29ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26461cbf4c204adbb639183a4a1d29ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35783 } } }
I20260620 12:19:45.564429 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:45.565285 3385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:45.571631 3385 catalog_manager.cc:1347] Generated new cluster ID: a7bec1e73b624c6f9380c336bdf86d1a
I20260620 12:19:45.571705 3385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:45.578011 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:45.573191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.0.98.62:35783)
0620 12:19:45.573317 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:45.573321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:45.573569 (+ 248us) server_negotiation.cc:407] Connection header received
0620 12:19:45.573734 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:45.573737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:45.573789 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:45.573876 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:45.574381 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:45.574921 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:45.575785 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:45.576043 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:45.576158 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:45.577081 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:45.577085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:45.577088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:45.577533 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:45.577561 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:45.577578 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:45.577681 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:45.577793 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 12:19:45.577852 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:19:45.589401 3385 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:45.590297 3385 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:45.608736 3385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26461cbf4c204adbb639183a4a1d29ba: Generated new TSK 0
I20260620 12:19:45.609691 3385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:19:45.623179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 3302
2026-06-20T12:19:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1439 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:45 dist-test-slave-9b3m krb5kdc[3403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:45 dist-test-slave-9b3m krb5kdc[3403](info): set up 2 sockets
Jun 20 12:19:45 dist-test-slave-9b3m krb5kdc[3403](info): commencing operation
krb5kdc: starting...
W20260620 12:19:46.689008 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:46 dist-test-slave-9b3m krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:46.815719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38253
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:46.924501 3416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:46.924780 3416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:46.924822 3416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:46.928926 3416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:46.928996 3416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:46.929016 3416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:46.929033 3416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:46.929066 3416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:46.934607 3416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:46.935725 3416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:46.936759 3416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:46.945544 3424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:46.945549 3422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:46.945715 3416 server_base.cc:1034] running on GCE node
W20260620 12:19:46.945561 3421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:46.946487 3416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:46.947619 3416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:46.948807 3416 hybrid_clock.cc:648] HybridClock initialized: now 1781957986948786 us; error 41 us; skew 500 ppm
Jun 20 12:19:46 dist-test-slave-9b3m krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:46.951622 3416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:46.952463 3416 webserver.cc:460] Webserver started at http://127.0.98.62:41197/ using document root <none> and password file <none>
I20260620 12:19:46.952989 3416 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:46.953094 3416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:46.953290 3416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:46.955237 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "1afec783e33b4e378b1655872078b41a"
format_stamp: "Formatted at 2026-06-20 12:19:46 on dist-test-slave-9b3m"
server_key: "1e0458ef95cde805a5c69a102e4df976"
server_key_iv: "eb278874e2222223e7a124a654f47ba0"
server_key_version: "encryptionkey@0"
I20260620 12:19:46.955750 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "1afec783e33b4e378b1655872078b41a"
format_stamp: "Formatted at 2026-06-20 12:19:46 on dist-test-slave-9b3m"
server_key: "1e0458ef95cde805a5c69a102e4df976"
server_key_iv: "eb278874e2222223e7a124a654f47ba0"
server_key_version: "encryptionkey@0"
I20260620 12:19:46.959422 3416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260620 12:19:46.961843 3431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:46.963274 3416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 12:19:46.963436 3416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "1afec783e33b4e378b1655872078b41a"
format_stamp: "Formatted at 2026-06-20 12:19:46 on dist-test-slave-9b3m"
server_key: "1e0458ef95cde805a5c69a102e4df976"
server_key_iv: "eb278874e2222223e7a124a654f47ba0"
server_key_version: "encryptionkey@0"
I20260620 12:19:46.963563 3416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:46.982234 3416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:46.985541 3416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:46.985746 3416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:46.992977 3416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38253
I20260620 12:19:46.993384 3483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38253 every 8 connection(s)
I20260620 12:19:46.993999 3416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:46.996850 3484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:47.001837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 3416
I20260620 12:19:47.001963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:47.002274 392 external_mini_cluster.cc:1428] Setting key 342e72c5bfe7c22f8fecb03a0467d35c
I20260620 12:19:47.002947 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a: Bootstrap starting.
I20260620 12:19:47.005575 3484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:47.006567 3484 log.cc:824] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:47.008805 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a: No bootstrap required, opened a new log
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:47.013478 3484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afec783e33b4e378b1655872078b41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38253 } }
I20260620 12:19:47.013744 3484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:47.013840 3484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1afec783e33b4e378b1655872078b41a, State: Initialized, Role: FOLLOWER
I20260620 12:19:47.014345 3484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afec783e33b4e378b1655872078b41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38253 } }
I20260620 12:19:47.014508 3484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:47.014606 3484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:47.014712 3484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:47.015637 3484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afec783e33b4e378b1655872078b41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38253 } }
I20260620 12:19:47.016002 3484 leader_election.cc:302] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1afec783e33b4e378b1655872078b41a; no voters:
I20260620 12:19:47.016382 3484 leader_election.cc:288] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:47.016492 3489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:47.016685 3489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [term 1 LEADER]: Becoming Leader. State: Replica: 1afec783e33b4e378b1655872078b41a, State: Running, Role: LEADER
I20260620 12:19:47.017015 3489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afec783e33b4e378b1655872078b41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38253 } }
I20260620 12:19:47.017520 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:47.003833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.0.98.62:38253)
0620 12:19:47.004244 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:47.004259 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:47.004294 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:19:47.004889 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:47.004906 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:47.005298 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:47.005573 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:47.006540 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.007573 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.008807 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.009383 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.009501 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:47.012384 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:47.012389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:47.012401 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:47.012427 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:47.014433 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.014941 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.014946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.014952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.015068 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.015372 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.015377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.015378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.015795 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:47.015967 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:47.016274 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 12:19:47.016583 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":133,"threads_started":1}
I20260620 12:19:47.017764 3484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:47.018733 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1afec783e33b4e378b1655872078b41a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afec783e33b4e378b1655872078b41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38253 } } }
I20260620 12:19:47.018793 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1afec783e33b4e378b1655872078b41a. Latest consensus state: current_term: 1 leader_uuid: "1afec783e33b4e378b1655872078b41a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afec783e33b4e378b1655872078b41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38253 } } }
I20260620 12:19:47.018894 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:47.018891 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:47.019615 3498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:47.022933 3498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:47.028600 3498 catalog_manager.cc:1347] Generated new cluster ID: 6c5d9e6b093c4778b2c7a5fd9161d25a
I20260620 12:19:47.028697 3498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:47.054392 3498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:47.055823 3498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:47.079651 3498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1afec783e33b4e378b1655872078b41a: Generated new TSK 0
I20260620 12:19:47.080442 3498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:47.112741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38253
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:47.245026 3510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:47.245282 3510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:47.245323 3510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:47.249425 3510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:47.249477 3510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:47.249568 3510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:47.254756 3510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:47.255754 3510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:47.256757 3510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:47.266626 3516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:47.266732 3510 server_base.cc:1034] running on GCE node
W20260620 12:19:47.266685 3515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:47.266923 3518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:47.267361 3510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:47.268122 3510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:47.269299 3510 hybrid_clock.cc:648] HybridClock initialized: now 1781957987269260 us; error 46 us; skew 500 ppm
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:47.272042 3510 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:47.272814 3510 webserver.cc:460] Webserver started at http://127.0.98.1:36543/ using document root <none> and password file <none>
I20260620 12:19:47.273371 3510 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:47.273435 3510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:47.273610 3510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:47.275513 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "9bbc786bca9d413f93a4a3b45f30cda5"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "3c3817bfa139cfffc8d591c4d9a37858"
server_key_iv: "fe0fe0a394e9f7457221f9b146588188"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.276011 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9bbc786bca9d413f93a4a3b45f30cda5"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "3c3817bfa139cfffc8d591c4d9a37858"
server_key_iv: "fe0fe0a394e9f7457221f9b146588188"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.279611 3510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:47.281939 3525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.283182 3510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:47.283398 3510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "9bbc786bca9d413f93a4a3b45f30cda5"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "3c3817bfa139cfffc8d591c4d9a37858"
server_key_iv: "fe0fe0a394e9f7457221f9b146588188"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.283506 3510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:47.294785 3510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:47.297883 3510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:47.298091 3510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:47.298666 3510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:47.299425 3510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:47.299468 3510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.299544 3510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:47.299576 3510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.307915 3510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46725
I20260620 12:19:47.307950 3638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46725 every 8 connection(s)
I20260620 12:19:47.308792 3510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:47.309303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 3510
I20260620 12:19:47.309439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:47.309751 392 external_mini_cluster.cc:1428] Setting key 16123d958b13e5d5e2ffbbeef3895272
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:47.323415 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:47.310753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35465 (local address 127.0.98.62:38253)
0620 12:19:47.310857 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:47.310870 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:47.311895 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 12:19:47.313389 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:47.313392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:47.313451 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:47.313546 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:47.314656 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.315543 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.316844 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.317335 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.317476 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:47.319946 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:47.319950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:47.319953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:47.319978 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:47.321437 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.321874 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.321877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.321878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.321946 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.322341 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.322346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.322348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.322517 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:47.322625 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:47.323133 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 12:19:47.323248 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 12:19:47.324155 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:47.311305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38253 (local address 127.0.98.1:35465)
0620 12:19:47.311762 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:47.311800 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:47.313188 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:47.313719 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:47.313725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:47.314058 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:47.314505 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:47.314518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.315690 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.315693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:47.316709 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:47.316719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.317495 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.317498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:47.317694 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:47.318305 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:47.318321 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:47.319773 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:47.321548 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:47.321552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:47.321563 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:47.321794 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:47.322098 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:47.322102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:47.322104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:47.322243 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:47.322621 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:47.322627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:47.322910 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 12:19:47.323095 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 12:19:47.323389 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260620 12:19:47.325464 3639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38253
I20260620 12:19:47.325721 3639 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:47.326181 3639 heartbeater.cc:511] Master 127.0.98.62:38253 requested a full tablet report, sending...
I20260620 12:19:47.327821 3448 ts_manager.cc:194] Registered new tserver with Master: 9bbc786bca9d413f93a4a3b45f30cda5 (127.0.98.1:46725)
I20260620 12:19:47.329501 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:47.340586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38253
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:47.447094 3644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:47.447336 3644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:47.447377 3644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:47.451385 3644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:47.451445 3644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:47.451503 3644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:47.456598 3644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:47.457618 3644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:47.458549 3644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:47.468005 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:47.467921 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:47.467921 3652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:47.468044 3644 server_base.cc:1034] running on GCE node
I20260620 12:19:47.468621 3644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:47.469357 3644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:47.470553 3644 hybrid_clock.cc:648] HybridClock initialized: now 1781957987470520 us; error 46 us; skew 500 ppm
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:47.473228 3644 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:47.474054 3644 webserver.cc:460] Webserver started at http://127.0.98.2:33571/ using document root <none> and password file <none>
I20260620 12:19:47.474597 3644 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:47.474682 3644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:47.474882 3644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:47.476826 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae90a8ac22e04a1ca36a910c6e2328b6"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "7da96563685c666209528f600be5e7aa"
server_key_iv: "7e4a5fea5a67cb5d68588c3b4dde64d4"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.477418 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae90a8ac22e04a1ca36a910c6e2328b6"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "7da96563685c666209528f600be5e7aa"
server_key_iv: "7e4a5fea5a67cb5d68588c3b4dde64d4"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.480855 3644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:19:47.483249 3659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.484472 3644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:47.484599 3644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "ae90a8ac22e04a1ca36a910c6e2328b6"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "7da96563685c666209528f600be5e7aa"
server_key_iv: "7e4a5fea5a67cb5d68588c3b4dde64d4"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.484711 3644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:47.499998 3644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:47.503055 3644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:47.503253 3644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:47.503824 3644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:47.504623 3644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:47.504702 3644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.504768 3644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:47.504803 3644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.513667 3644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36889
I20260620 12:19:47.513693 3772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36889 every 8 connection(s)
I20260620 12:19:47.514531 3644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:47.516319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 3644
I20260620 12:19:47.516470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:47.516822 392 external_mini_cluster.cc:1428] Setting key 57834f4942764c482378a54a21cfcd80
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:19:47.529973 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:47.516256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45167 (local address 127.0.98.62:38253)
0620 12:19:47.516403 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:47.516408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:47.517419 (+ 1011us) server_negotiation.cc:407] Connection header received
0620 12:19:47.519226 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:47.519230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:47.519279 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:47.519349 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:47.520722 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.521616 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.523023 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.523450 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.523597 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:47.526176 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:47.526179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:47.526181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:47.526204 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:47.527824 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.528377 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.528380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.528382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.528463 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.528784 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.528787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.528788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.529012 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:47.529177 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:47.529688 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 12:19:47.529807 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:19:47.531054 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:47.516712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38253 (local address 127.0.98.2:45167)
0620 12:19:47.517224 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:47.517303 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:47.518984 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:47.519543 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:47.519554 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:47.520079 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:47.520579 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:47.520593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.521765 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.521769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:47.522876 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:47.522887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.523639 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.523642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:47.523784 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:47.524382 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:47.524398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:47.526000 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:47.527946 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:47.527954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:47.527999 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:47.528257 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:47.528560 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:47.528562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:47.528563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:47.528690 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:47.529181 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:47.529190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:47.529433 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 12:19:47.529632 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 12:19:47.529959 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260620 12:19:47.532302 3773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38253
I20260620 12:19:47.532568 3773 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:47.533258 3773 heartbeater.cc:511] Master 127.0.98.62:38253 requested a full tablet report, sending...
I20260620 12:19:47.534431 3448 ts_manager.cc:194] Registered new tserver with Master: ae90a8ac22e04a1ca36a910c6e2328b6 (127.0.98.2:36889)
I20260620 12:19:47.535223 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:19:47.545233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38253
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:47.655920 3778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:47.656144 3778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:47.656183 3778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:47.660130 3778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:47.660181 3778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:47.660274 3778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:47.665241 3778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:47.666183 3778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:47.667129 3778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:47.676461 3783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:47.676462 3784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:47.676563 3786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:47.676963 3778 server_base.cc:1034] running on GCE node
I20260620 12:19:47.677392 3778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:47.678061 3778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:47.679282 3778 hybrid_clock.cc:648] HybridClock initialized: now 1781957987679241 us; error 32 us; skew 500 ppm
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:47.682260 3778 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:47.683141 3778 webserver.cc:460] Webserver started at http://127.0.98.3:41889/ using document root <none> and password file <none>
I20260620 12:19:47.683666 3778 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:47.683727 3778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:47.683888 3778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:47.686185 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "dd1624ad15f34f59b763a9fa648f1dee"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "9c35d173e9e79444e8e4ad50065efeb9"
server_key_iv: "75cb7de7e174978a48733b5edce7f0c6"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.686753 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dd1624ad15f34f59b763a9fa648f1dee"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "9c35d173e9e79444e8e4ad50065efeb9"
server_key_iv: "75cb7de7e174978a48733b5edce7f0c6"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.690842 3778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:19:47.693331 3793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.694571 3778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:47.694725 3778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "dd1624ad15f34f59b763a9fa648f1dee"
format_stamp: "Formatted at 2026-06-20 12:19:47 on dist-test-slave-9b3m"
server_key: "9c35d173e9e79444e8e4ad50065efeb9"
server_key_iv: "75cb7de7e174978a48733b5edce7f0c6"
server_key_version: "encryptionkey@0"
I20260620 12:19:47.694840 3778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:47.723896 3778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:47.727075 3778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:47.727299 3778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:47.727923 3778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:47.728776 3778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:47.728855 3778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.728922 3778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:47.728976 3778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:47.738379 3778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40539
I20260620 12:19:47.738432 3906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40539 every 8 connection(s)
I20260620 12:19:47.739349 3778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:47.741127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 3778
I20260620 12:19:47.741290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:47.741590 392 external_mini_cluster.cc:1428] Setting key b61ffb59c3cdbe6ec2ce877a2c74d493
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:47.753239 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:47.741364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35093 (local address 127.0.98.62:38253)
0620 12:19:47.741518 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:47.741523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:47.742576 (+ 1053us) server_negotiation.cc:407] Connection header received
0620 12:19:47.744136 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:47.744139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:47.744183 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:47.744249 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:47.745404 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.745926 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.747172 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.747381 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.747461 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:47.749883 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:47.749886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:47.749888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:47.749909 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:47.751196 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.751678 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.751681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.751682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.751767 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.752170 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.752173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.752175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.752376 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:47.752473 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:47.752939 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 12:19:47.753102 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:19:47.754007 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:47.741886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38253 (local address 127.0.98.3:35093)
0620 12:19:47.742420 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:47.742482 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:47.743950 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:47.744383 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:47.744389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:47.744735 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:47.745281 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:47.745296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.746066 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.746069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:47.747061 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:47.747070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.747521 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.747525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:47.747671 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:47.748200 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:47.748217 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:47.749715 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:47.751297 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:47.751301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:47.751312 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:47.751556 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:47.751879 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:47.751881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:47.751883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:47.752059 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:47.752484 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:47.752489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:47.752715 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 12:19:47.752865 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 12:19:47.753194 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":156,"threads_started":1}
I20260620 12:19:47.755219 3907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38253
I20260620 12:19:47.755456 3907 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:47.755927 3907 heartbeater.cc:511] Master 127.0.98.62:38253 requested a full tablet report, sending...
I20260620 12:19:47.756945 3448 ts_manager.cc:194] Registered new tserver with Master: dd1624ad15f34f59b763a9fa648f1dee (127.0.98.3:40539)
I20260620 12:19:47.757560 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35093
I20260620 12:19:47.766960 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957987, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957987, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:47.793385 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:47.784011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.0.98.62:38253)
0620 12:19:47.784178 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:47.784183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:47.784377 (+ 194us) server_negotiation.cc:407] Connection header received
0620 12:19:47.784443 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:47.784446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:47.784492 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:47.784573 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:47.785441 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.786017 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.787124 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:47.787373 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:47.787467 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:47.790121 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:47.790125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:47.790127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:47.790155 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:47.791494 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.791954 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.791958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.791961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.792050 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:47.792343 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:47.792348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:47.792350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:47.792603 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:47.792756 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:47.793070 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 12:19:47.793208 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 12:19:47.793828 3448 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38170
I20260620 12:19:47.794749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 3510
I20260620 12:19:47.800310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 3644
I20260620 12:19:47.805601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 3778
I20260620 12:19:47.810796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 3416
2026-06-20T12:19:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2190 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3919](info): set up 2 sockets
Jun 20 12:19:47 dist-test-slave-9b3m krb5kdc[3919](info): commencing operation
krb5kdc: starting...
W20260620 12:19:48.885496 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:48 dist-test-slave-9b3m krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:49.005931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:49.112792 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:49.113021 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:49.113137 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:49.117009 3932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:49.117074 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:49.117094 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:49.117111 3932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:49.117172 3932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:49.122135 3932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:49.123078 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:49.124045 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:49.132243 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:49.132272 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:49.132278 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:49.132565 3932 server_base.cc:1034] running on GCE node
I20260620 12:19:49.133141 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:49.134249 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:49.135440 3932 hybrid_clock.cc:648] HybridClock initialized: now 1781957989135427 us; error 26 us; skew 500 ppm
Jun 20 12:19:49 dist-test-slave-9b3m krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:49.138172 3932 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:49.139006 3932 webserver.cc:460] Webserver started at http://127.0.98.62:44747/ using document root <none> and password file <none>
I20260620 12:19:49.139550 3932 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:49.139639 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:49.139858 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:49.141907 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "dc4f8f081dd74818ac0dfe1d292cc857"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "c70473275794efe3945f9e41c05e7cc2"
server_key_iv: "2f48a84b5dc65c29715ac344e0437594"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.142450 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc4f8f081dd74818ac0dfe1d292cc857"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "c70473275794efe3945f9e41c05e7cc2"
server_key_iv: "2f48a84b5dc65c29715ac344e0437594"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.146034 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:49.148278 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:49.149559 3932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:49.149698 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "dc4f8f081dd74818ac0dfe1d292cc857"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "c70473275794efe3945f9e41c05e7cc2"
server_key_iv: "2f48a84b5dc65c29715ac344e0437594"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.149801 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:49.174479 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:49.177361 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:49.177569 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:49.184350 3932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35011
I20260620 12:19:49.184429 3999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35011 every 8 connection(s)
I20260620 12:19:49.185323 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:49.187919 4000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:49.191965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 3932
I20260620 12:19:49.192118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:49.192381 392 external_mini_cluster.cc:1428] Setting key ed2e590d7dbec5c9be75b46bea7456e8
I20260620 12:19:49.193656 4000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857: Bootstrap starting.
I20260620 12:19:49.195963 4000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:49.196712 4000 log.cc:824] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:49.198457 4000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857: No bootstrap required, opened a new log
Jun 20 12:19:49 dist-test-slave-9b3m krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:49.203197 4000 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4f8f081dd74818ac0dfe1d292cc857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } }
I20260620 12:19:49.203474 4000 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:49.203588 4000 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4f8f081dd74818ac0dfe1d292cc857, State: Initialized, Role: FOLLOWER
I20260620 12:19:49.204079 4000 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4f8f081dd74818ac0dfe1d292cc857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } }
I20260620 12:19:49.204244 4000 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:49.204308 4000 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:49.204411 4000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:49.205299 4000 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4f8f081dd74818ac0dfe1d292cc857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } }
I20260620 12:19:49.205675 4000 leader_election.cc:302] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc4f8f081dd74818ac0dfe1d292cc857; no voters:
I20260620 12:19:49.206005 4000 leader_election.cc:288] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:49.206065 4005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:49.206254 4005 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [term 1 LEADER]: Becoming Leader. State: Replica: dc4f8f081dd74818ac0dfe1d292cc857, State: Running, Role: LEADER
I20260620 12:19:49.206511 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:49.193685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.0.98.62:35011)
0620 12:19:49.194098 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:49.194114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:49.194147 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:19:49.194737 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:49.194752 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:49.195066 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:49.195326 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:49.196201 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.197273 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.198098 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.198577 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.198714 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:49.201443 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:49.201452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:49.201472 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:49.201511 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:49.203429 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:49.203955 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:49.203960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:49.203965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:49.204099 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:49.204443 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:49.204447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:49.204449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:49.204846 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:49.205114 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:49.205399 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 12:19:49.205656 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260620 12:19:49.206578 4005 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4f8f081dd74818ac0dfe1d292cc857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } }
I20260620 12:19:49.207237 4000 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:49.208302 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc4f8f081dd74818ac0dfe1d292cc857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4f8f081dd74818ac0dfe1d292cc857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } } }
I20260620 12:19:49.208418 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:49.208851 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc4f8f081dd74818ac0dfe1d292cc857. Latest consensus state: current_term: 1 leader_uuid: "dc4f8f081dd74818ac0dfe1d292cc857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4f8f081dd74818ac0dfe1d292cc857" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } } }
I20260620 12:19:49.209009 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857 [sys.catalog]: This master's current role is: LEADER
W20260620 12:19:49.211988 4021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:19:49.212073 4021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:19:49.212146 4014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:49.212833 4014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:49.218420 4014 catalog_manager.cc:1347] Generated new cluster ID: bd2ccffe28914c90ac5cb3bd4b78419f
I20260620 12:19:49.218478 4014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:49.235060 4014 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:49.236009 4014 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:49.241215 4014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc4f8f081dd74818ac0dfe1d292cc857: Generated new TSK 0
I20260620 12:19:49.241888 4014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:49.269750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:49.377023 4026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:49.377345 4026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:49.377450 4026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:49.381405 4026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:49.381479 4026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:49.381557 4026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:49.386586 4026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:49.387616 4026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:49.388597 4026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:49.397789 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:49.397840 4031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:49.397843 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:49.398165 4026 server_base.cc:1034] running on GCE node
I20260620 12:19:49.398533 4026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:49.399163 4026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:49.400344 4026 hybrid_clock.cc:648] HybridClock initialized: now 1781957989400331 us; error 24 us; skew 500 ppm
Jun 20 12:19:49 dist-test-slave-9b3m krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:49.402999 4026 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:49.403777 4026 webserver.cc:460] Webserver started at http://127.0.98.1:34253/ using document root <none> and password file <none>
I20260620 12:19:49.404318 4026 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:49.404414 4026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:49.404635 4026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:49.406628 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb206b8284a9496db0cfcb3664440ef2"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "45a09d789ebb4b6439dff8c2cfdf3311"
server_key_iv: "012298e20fbdfdc1fd6af7b603c98de4"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.407228 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb206b8284a9496db0cfcb3664440ef2"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "45a09d789ebb4b6439dff8c2cfdf3311"
server_key_iv: "012298e20fbdfdc1fd6af7b603c98de4"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.410801 4026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 12:19:49.413173 4041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:49.414347 4026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:49.414479 4026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "bb206b8284a9496db0cfcb3664440ef2"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "45a09d789ebb4b6439dff8c2cfdf3311"
server_key_iv: "012298e20fbdfdc1fd6af7b603c98de4"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.414585 4026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:49.434825 4026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:49.437594 4026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:49.437788 4026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:49.438349 4026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:49.439159 4026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:49.439230 4026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:49.439299 4026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:49.439352 4026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:49.448625 4026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45407
I20260620 12:19:49.448654 4154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45407 every 8 connection(s)
I20260620 12:19:49.449669 4026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:49.455822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 4026
I20260620 12:19:49.455967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:49.456346 392 external_mini_cluster.cc:1428] Setting key 6f8ab752b491614e13f5d2e8e5f5193b
Jun 20 12:19:49 dist-test-slave-9b3m krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:49.464198 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:49.451801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55199 (local address 127.0.98.62:35011)
0620 12:19:49.451974 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:49.451978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:49.452838 (+ 860us) server_negotiation.cc:407] Connection header received
0620 12:19:49.454306 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:49.454309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:49.454361 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:49.454454 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:49.455608 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.456568 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.457731 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.458139 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.458285 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:49.460619 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:49.460623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:49.460626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:49.460653 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:49.462224 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:49.462721 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:49.462725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:49.462727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:49.462808 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:49.463142 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:49.463146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:49.463148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:49.463323 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:49.463438 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:49.463920 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 12:19:49.464040 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 12:19:49.465003 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:49.452175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35011 (local address 127.0.98.1:55199)
0620 12:19:49.452696 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:49.452737 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:49.454106 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:49.454605 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:49.454612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:49.454983 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:49.455443 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:49.455455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.456703 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.456706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:49.457633 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:49.457641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.458304 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.458306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:49.458444 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:49.459045 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:49.459061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:49.460457 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:49.462337 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:49.462343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:49.462358 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:49.462614 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:49.462904 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:49.462907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:49.462909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:49.463044 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:49.463437 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:49.463444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:49.463688 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 12:19:49.463886 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 12:19:49.464193 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":114,"threads_started":1}
I20260620 12:19:49.466305 4155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35011
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:19:49.466585 4155 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:49.467154 4155 heartbeater.cc:511] Master 127.0.98.62:35011 requested a full tablet report, sending...
I20260620 12:19:49.468823 3964 ts_manager.cc:194] Registered new tserver with Master: bb206b8284a9496db0cfcb3664440ef2 (127.0.98.1:45407)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:49.470222 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:49.484747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:49.603876 4160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:49.604233 4160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:49.604310 4160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:49.608786 4160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:49.608848 4160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:49.608954 4160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:49.614602 4160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:49.615672 4160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:49.616761 4160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:19:49.626605 4160 server_base.cc:1034] running on GCE node
W20260620 12:19:49.626531 4166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:49.626551 4168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:49.626551 4165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:49.627242 4160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:49.627926 4160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:49.629158 4160 hybrid_clock.cc:648] HybridClock initialized: now 1781957989629134 us; error 38 us; skew 500 ppm
Jun 20 12:19:49 dist-test-slave-9b3m krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:49.631946 4160 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:49.632879 4160 webserver.cc:460] Webserver started at http://127.0.98.2:36639/ using document root <none> and password file <none>
I20260620 12:19:49.633479 4160 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:49.633548 4160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:49.633718 4160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:49.635707 4160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "2539fb773a184f7c947c4c7d65dd4b08"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "1fd6ee2c13c1b4f7766103e8d270f5e2"
server_key_iv: "695685efb296a3ec3366c4a34c31a44f"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.636234 4160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2539fb773a184f7c947c4c7d65dd4b08"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "1fd6ee2c13c1b4f7766103e8d270f5e2"
server_key_iv: "695685efb296a3ec3366c4a34c31a44f"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.639891 4160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:19:49.642359 4175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:49.643682 4160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:49.643777 4160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "2539fb773a184f7c947c4c7d65dd4b08"
format_stamp: "Formatted at 2026-06-20 12:19:49 on dist-test-slave-9b3m"
server_key: "1fd6ee2c13c1b4f7766103e8d270f5e2"
server_key_iv: "695685efb296a3ec3366c4a34c31a44f"
server_key_version: "encryptionkey@0"
I20260620 12:19:49.643918 4160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:49.669863 4160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:49.673245 4160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:49.673456 4160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:49.674111 4160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:49.674954 4160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:49.674999 4160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:49.675069 4160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:49.675107 4160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:49.685403 4160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41565
I20260620 12:19:49.685459 4288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41565 every 8 connection(s)
I20260620 12:19:49.686658 4160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:49.691074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 4160
I20260620 12:19:49.691223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:49.691509 392 external_mini_cluster.cc:1428] Setting key 35fcc40639eb9edd5c4b29c2f85adfc8
Jun 20 12:19:49 dist-test-slave-9b3m krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:19:49.702347 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:49.688598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50545 (local address 127.0.98.62:35011)
0620 12:19:49.688732 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:49.688736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:49.689664 (+ 928us) server_negotiation.cc:407] Connection header received
0620 12:19:49.691082 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:49.691085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:49.691141 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:49.691220 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:49.692710 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.693398 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.695106 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.695550 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.695697 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:49.698271 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:49.698274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:49.698277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:49.698305 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:49.699866 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:49.700515 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:49.700519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:49.700521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:49.700618 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:49.701053 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:49.701079 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:49.701081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:49.701286 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:49.701373 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:49.702070 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 12:19:49.702212 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:19:49.703197 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:49.688978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35011 (local address 127.0.98.2:50545)
0620 12:19:49.689469 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:49.689530 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:49.690869 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:49.691536 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:49.691543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:49.692001 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:49.692563 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:49.692577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.693531 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.693538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:49.694960 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:49.694970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:49.695711 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:49.695714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:49.695890 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:49.696545 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:49.696561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:49.698090 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:49.699992 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:49.699998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:49.700015 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:49.700386 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:49.700732 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:49.700737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:49.700742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:49.700932 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:49.701407 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:49.701415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:49.701758 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 12:19:49.701987 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 12:19:49.702363 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
I20260620 12:19:49.704417 4289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35011
I20260620 12:19:49.704684 4289 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:49.705348 4289 heartbeater.cc:511] Master 127.0.98.62:35011 requested a full tablet report, sending...
I20260620 12:19:49.706363 3964 ts_manager.cc:194] Registered new tserver with Master: 2539fb773a184f7c947c4c7d65dd4b08 (127.0.98.2:41565)
I20260620 12:19:49.707206 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260620 12:19:49.976610 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.283s user 0.002s sys 0.002s
I20260620 12:19:49.977249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:50.088845 4294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:50.089164 4294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:50.089206 4294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:50.093319 4294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:50.093372 4294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:50.093427 4294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:50.098379 4294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:50.099327 4294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:50.100242 4294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:50.110128 4300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:50.110152 4299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:50.110169 4302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:50.110332 4294 server_base.cc:1034] running on GCE node
I20260620 12:19:50.110682 4294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:50.111266 4294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:50.112429 4294 hybrid_clock.cc:648] HybridClock initialized: now 1781957990112416 us; error 28 us; skew 500 ppm
Jun 20 12:19:50 dist-test-slave-9b3m krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:50.115096 4294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:50.115935 4294 webserver.cc:460] Webserver started at http://127.0.98.3:35085/ using document root <none> and password file <none>
I20260620 12:19:50.116420 4294 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:50.116477 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:50.116626 4294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:50.118764 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed6d203c5a454c88bf331eb95c83820a"
format_stamp: "Formatted at 2026-06-20 12:19:50 on dist-test-slave-9b3m"
server_key: "247579f7418053e8b6acfa4226bbf242"
server_key_iv: "e856da75c4581c3f62ad4b1f18f77c14"
server_key_version: "encryptionkey@0"
I20260620 12:19:50.119266 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed6d203c5a454c88bf331eb95c83820a"
format_stamp: "Formatted at 2026-06-20 12:19:50 on dist-test-slave-9b3m"
server_key: "247579f7418053e8b6acfa4226bbf242"
server_key_iv: "e856da75c4581c3f62ad4b1f18f77c14"
server_key_version: "encryptionkey@0"
I20260620 12:19:50.122725 4294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:50.124969 4309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:50.126129 4294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:50.126260 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "ed6d203c5a454c88bf331eb95c83820a"
format_stamp: "Formatted at 2026-06-20 12:19:50 on dist-test-slave-9b3m"
server_key: "247579f7418053e8b6acfa4226bbf242"
server_key_iv: "e856da75c4581c3f62ad4b1f18f77c14"
server_key_version: "encryptionkey@0"
I20260620 12:19:50.126367 4294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:50.141759 4294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:50.144572 4294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:50.144764 4294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:50.145370 4294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:50.146143 4294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:50.146214 4294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:50.146281 4294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:50.146315 4294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:50.155200 4294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45159
I20260620 12:19:50.155220 4422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45159 every 8 connection(s)
I20260620 12:19:50.156071 4294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:50.163271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 4294
I20260620 12:19:50.163414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:50.163702 392 external_mini_cluster.cc:1428] Setting key 0e5f53dd6baa79c29c86d0680c91d868
Jun 20 12:19:50 dist-test-slave-9b3m krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:50.169420 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:50.157859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58975 (local address 127.0.98.62:35011)
0620 12:19:50.157997 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:50.158001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:50.158759 (+ 758us) server_negotiation.cc:407] Connection header received
0620 12:19:50.160093 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:50.160097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:50.160153 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:50.160258 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:50.161419 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:50.161948 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:50.163063 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:50.163286 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:50.163393 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:50.166237 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:50.166241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:50.166243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:50.166266 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:50.167623 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:50.168060 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:50.168062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:50.168063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:50.168132 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:50.168420 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:50.168422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:50.168423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:50.168575 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:50.168647 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:50.169156 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 12:19:50.169280 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:19:50.170187 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:50.158201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35011 (local address 127.0.98.3:58975)
0620 12:19:50.158602 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:50.158647 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:50.159893 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:50.160436 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:50.160443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:50.160770 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:50.161279 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:50.161293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:50.162068 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:50.162073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:50.162926 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:50.162935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:50.163425 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:50.163429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:50.163622 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:50.164269 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:50.164292 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:50.166081 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:50.167720 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:50.167725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:50.167735 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:50.167977 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:50.168207 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:50.168209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:50.168211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:50.168345 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:50.168656 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:50.168661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:50.168869 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 12:19:50.169107 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 12:19:50.169415 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260620 12:19:50.171203 4423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35011
I20260620 12:19:50.171402 4423 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:50.171793 4423 heartbeater.cc:511] Master 127.0.98.62:35011 requested a full tablet report, sending...
I20260620 12:19:50.172669 3964 ts_manager.cc:194] Registered new tserver with Master: ed6d203c5a454c88bf331eb95c83820a (127.0.98.3:45159)
I20260620 12:19:50.173270 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58975
I20260620 12:19:50.177659 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 12:19:50 dist-test-slave-9b3m krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:19:50 dist-test-slave-9b3m krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:50.202291 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:50.193412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.0.98.1:45407)
0620 12:19:50.193700 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:50.193708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:50.193735 (+ 27us) server_negotiation.cc:407] Connection header received
0620 12:19:50.193833 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:50.193839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:50.194045 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:50.194173 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:50.194795 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:50.195766 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:50.196563 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:50.196912 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:50.196996 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:50.199307 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:50.199315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:50.199326 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:50.199347 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:50.200755 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:50.201168 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:50.201173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:50.201174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:50.201256 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:50.201514 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:50.201516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:50.201518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:50.201728 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:50.201883 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:50.202033 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 12:19:50.202147 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
Jun 20 12:19:50 dist-test-slave-9b3m krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:50.212525 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:50.204795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.0.98.62:35011)
0620 12:19:50.204919 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:50.204922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:50.205107 (+ 185us) server_negotiation.cc:407] Connection header received
0620 12:19:50.205221 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:50.205224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:50.205262 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:50.205331 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:50.205876 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:50.206358 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:50.207096 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:50.207338 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:50.207415 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:50.209685 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:50.209688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:50.209690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:50.209710 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:50.211135 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:50.211485 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:50.211488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:50.211489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:50.211556 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:50.211782 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:50.211784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:50.211785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:50.211986 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:50.212085 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:50.212289 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 12:19:50.212419 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 12:19:50.212900 3964 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44162
I20260620 12:19:50.213474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 4026
I20260620 12:19:50.218714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 4160
I20260620 12:19:50.223853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 4294
I20260620 12:19:50.228842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 3932
2026-06-20T12:19:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2417 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:50 dist-test-slave-9b3m krb5kdc[4435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:50 dist-test-slave-9b3m krb5kdc[4435](info): set up 2 sockets
Jun 20 12:19:50 dist-test-slave-9b3m krb5kdc[4435](info): commencing operation
krb5kdc: starting...
W20260620 12:19:51.269392 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:51 dist-test-slave-9b3m krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:51.391703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:51.500998 4448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:51.501268 4448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:51.501312 4448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:51.505245 4448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:51.505298 4448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:51.505321 4448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:51.505378 4448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:51.505415 4448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:51.510617 4448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:51.511598 4448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:51.512593 4448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:51.520864 4454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:51.520928 4448 server_base.cc:1034] running on GCE node
W20260620 12:19:51.520860 4453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:51.520864 4456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:51.521682 4448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:51.522753 4448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:51.523936 4448 hybrid_clock.cc:648] HybridClock initialized: now 1781957991523908 us; error 43 us; skew 500 ppm
Jun 20 12:19:51 dist-test-slave-9b3m krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:51.530596 4448 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:51.531267 4448 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:51.531327 4448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:51.531481 4448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:51.533514 4448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "3c13a1c601a54ecebfc6d3c4f2093e83"
format_stamp: "Formatted at 2026-06-20 12:19:51 on dist-test-slave-9b3m"
server_key: "adb7b897892b3aafa79e7296b2fdd698"
server_key_iv: "bca456717bb0a80cd58276d6e295fa3e"
server_key_version: "encryptionkey@0"
I20260620 12:19:51.534008 4448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c13a1c601a54ecebfc6d3c4f2093e83"
format_stamp: "Formatted at 2026-06-20 12:19:51 on dist-test-slave-9b3m"
server_key: "adb7b897892b3aafa79e7296b2fdd698"
server_key_iv: "bca456717bb0a80cd58276d6e295fa3e"
server_key_version: "encryptionkey@0"
I20260620 12:19:51.537509 4448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:19:51.539817 4462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:51.541018 4448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:51.541188 4448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "3c13a1c601a54ecebfc6d3c4f2093e83"
format_stamp: "Formatted at 2026-06-20 12:19:51 on dist-test-slave-9b3m"
server_key: "adb7b897892b3aafa79e7296b2fdd698"
server_key_iv: "bca456717bb0a80cd58276d6e295fa3e"
server_key_version: "encryptionkey@0"
I20260620 12:19:51.541272 4448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:51.562600 4448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:51.565793 4448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:51.565937 4448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:51.572751 4448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37337
I20260620 12:19:51.573009 4514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37337 every 8 connection(s)
I20260620 12:19:51.573454 4448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:51.576115 4515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:51.577710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 4448
I20260620 12:19:51.577836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:51.578120 392 external_mini_cluster.cc:1428] Setting key 879d92bda30110858db458bc98d7fcb2
I20260620 12:19:51.582334 4515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83: Bootstrap starting.
I20260620 12:19:51.584846 4515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:51.585829 4515 log.cc:824] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:19:51 dist-test-slave-9b3m krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:51.588142 4515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83: No bootstrap required, opened a new log
I20260620 12:19:51.592095 4515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a1c601a54ecebfc6d3c4f2093e83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37337 } }
I20260620 12:19:51.592322 4515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:51.592382 4515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c13a1c601a54ecebfc6d3c4f2093e83, State: Initialized, Role: FOLLOWER
I20260620 12:19:51.592780 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:51.579384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.0.98.62:37337)
0620 12:19:51.579787 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:51.579805 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:51.579841 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:19:51.580435 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:51.580451 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:51.580767 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:51.581033 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:51.581936 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:51.582953 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:51.584120 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:51.584779 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:51.584947 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:51.588088 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:51.588093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:51.588105 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:51.588130 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:51.589921 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:51.590440 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:51.590444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:51.590449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:51.590551 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:51.590822 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:51.590824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:51.590825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:51.591176 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:51.591334 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:51.591613 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 12:19:51.591914 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":126,"threads_started":1}
I20260620 12:19:51.592831 4515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a1c601a54ecebfc6d3c4f2093e83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37337 } }
I20260620 12:19:51.592991 4515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:51.593084 4515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:51.593214 4515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:51.594146 4515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a1c601a54ecebfc6d3c4f2093e83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37337 } }
I20260620 12:19:51.594465 4515 leader_election.cc:302] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c13a1c601a54ecebfc6d3c4f2093e83; no voters:
I20260620 12:19:51.594771 4515 leader_election.cc:288] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:51.594859 4520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:51.595181 4520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [term 1 LEADER]: Becoming Leader. State: Replica: 3c13a1c601a54ecebfc6d3c4f2093e83, State: Running, Role: LEADER
I20260620 12:19:51.595508 4520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a1c601a54ecebfc6d3c4f2093e83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37337 } }
I20260620 12:19:51.596123 4515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:51.597174 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c13a1c601a54ecebfc6d3c4f2093e83. Latest consensus state: current_term: 1 leader_uuid: "3c13a1c601a54ecebfc6d3c4f2093e83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a1c601a54ecebfc6d3c4f2093e83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37337 } } }
I20260620 12:19:51.597316 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:51.597635 4522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c13a1c601a54ecebfc6d3c4f2093e83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a1c601a54ecebfc6d3c4f2093e83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37337 } } }
I20260620 12:19:51.597724 4522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:51.597731 4529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:51.600746 4529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:51.606868 4529 catalog_manager.cc:1347] Generated new cluster ID: 4553c2f9158a46ed86c2136ba1e4bb84
I20260620 12:19:51.606935 4529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:51.624990 4529 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:51.626273 4529 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:51.634894 4529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83: Generated new TSK 0
I20260620 12:19:51.635664 4529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:51.664893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37337
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:51.772609 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:51.772877 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:51.772951 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:51.776973 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:51.777078 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:51.777179 4541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:51.782405 4541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:51.783380 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:51.784368 4541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:51.793421 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:51.793606 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:51.793701 4541 server_base.cc:1034] running on GCE node
W20260620 12:19:51.793609 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:51.794209 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:51.794852 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:51.796046 4541 hybrid_clock.cc:648] HybridClock initialized: now 1781957991796021 us; error 56 us; skew 500 ppm
Jun 20 12:19:51 dist-test-slave-9b3m krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:51.799103 4541 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:51.799816 4541 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:51.799908 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:51.800128 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:51.802181 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "55a49b238654459ab647dbb11232e110"
format_stamp: "Formatted at 2026-06-20 12:19:51 on dist-test-slave-9b3m"
server_key: "05e8e15fd2c0e447d3f69834018488f9"
server_key_iv: "87ad9a66a8d3fc5fcbe5715f9315a581"
server_key_version: "encryptionkey@0"
I20260620 12:19:51.802733 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "55a49b238654459ab647dbb11232e110"
format_stamp: "Formatted at 2026-06-20 12:19:51 on dist-test-slave-9b3m"
server_key: "05e8e15fd2c0e447d3f69834018488f9"
server_key_iv: "87ad9a66a8d3fc5fcbe5715f9315a581"
server_key_version: "encryptionkey@0"
I20260620 12:19:51.806243 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:51.808460 4555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:51.809700 4541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:51.809831 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "55a49b238654459ab647dbb11232e110"
format_stamp: "Formatted at 2026-06-20 12:19:51 on dist-test-slave-9b3m"
server_key: "05e8e15fd2c0e447d3f69834018488f9"
server_key_iv: "87ad9a66a8d3fc5fcbe5715f9315a581"
server_key_version: "encryptionkey@0"
I20260620 12:19:51.809935 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:51.831001 4541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:51.834290 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:51.834484 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:51.835043 4541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:51.835842 4541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:51.835911 4541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:51.835981 4541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:51.836032 4541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:51.844692 4541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34963
I20260620 12:19:51.844712 4668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34963 every 8 connection(s)
I20260620 12:19:51.845378 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:51.850951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 4541
I20260620 12:19:51.851105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:51.851409 392 external_mini_cluster.cc:1428] Setting key 2fc2cb75f8eace6df9dcb21e2baea2d3
Jun 20 12:19:51 dist-test-slave-9b3m krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:51.860020 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:51.847003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42731 (local address 127.0.98.62:37337)
0620 12:19:51.847151 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:51.847156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:51.847958 (+ 802us) server_negotiation.cc:407] Connection header received
0620 12:19:51.849309 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:51.849312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:51.849357 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:51.849433 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:51.850590 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:51.851289 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:51.852770 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:51.853036 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:51.853175 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:51.856295 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:51.856298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:51.856300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:51.856323 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:51.857764 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:51.858390 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:51.858393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:51.858394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:51.858462 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:51.858896 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:51.858899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:51.858901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:51.859098 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:51.859187 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:51.859798 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 12:19:51.859905 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:19:51.861218 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:51.847402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37337 (local address 127.0.98.1:42731)
0620 12:19:51.847815 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:51.847856 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:51.849032 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:51.849585 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:51.849591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:51.849917 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:51.850382 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:51.850395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:51.851423 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:51.851429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:51.852632 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:51.852643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:51.853235 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:51.853240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:51.853468 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:51.854106 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:51.854131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:51.856140 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:51.857883 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:51.857892 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:51.857910 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:51.858289 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:51.858554 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:51.858560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:51.858564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:51.858769 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:51.859221 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:51.859229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:51.859533 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 12:19:51.859762 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 12:19:51.860099 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:19:51.862507 4669 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37337
I20260620 12:19:51.862716 4669 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:51.863112 4669 heartbeater.cc:511] Master 127.0.98.62:37337 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:51.864822 4479 ts_manager.cc:194] Registered new tserver with Master: 55a49b238654459ab647dbb11232e110 (127.0.98.1:34963)
I20260620 12:19:51.866211 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:51.879575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37337
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:51.989915 4674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:51.990159 4674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:51.990204 4674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:51.994170 4674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:51.994253 4674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:51.994343 4674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:51.999643 4674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:52.000568 4674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:52.001582 4674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:52.010762 4682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:52.010790 4679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:52.010991 4674 server_base.cc:1034] running on GCE node
W20260620 12:19:52.010710 4680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:52.011544 4674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:52.012122 4674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:52.013278 4674 hybrid_clock.cc:648] HybridClock initialized: now 1781957992013259 us; error 34 us; skew 500 ppm
Jun 20 12:19:52 dist-test-slave-9b3m krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:52.016363 4674 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:52.017208 4674 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:52.017274 4674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:52.017481 4674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:52.019412 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "0a4235a825354fa9a1310d78d00d8456"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "59edc2f76869879bd6c80cc094584da0"
server_key_iv: "0d2f98d2938498ad9c7f8883f29e759d"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.019948 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0a4235a825354fa9a1310d78d00d8456"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "59edc2f76869879bd6c80cc094584da0"
server_key_iv: "0d2f98d2938498ad9c7f8883f29e759d"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.023383 4674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:52.025768 4688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:52.026926 4674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:52.027015 4674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "0a4235a825354fa9a1310d78d00d8456"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "59edc2f76869879bd6c80cc094584da0"
server_key_iv: "0d2f98d2938498ad9c7f8883f29e759d"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.027138 4674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:52.045749 4674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:52.048964 4674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:52.049209 4674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:52.049870 4674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:52.051054 4674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:52.051110 4674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:52.051182 4674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:52.051220 4674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:52.060535 4674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46243
I20260620 12:19:52.060557 4801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46243 every 8 connection(s)
I20260620 12:19:52.061270 4674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:52.065410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 4674
I20260620 12:19:52.065544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:52.065958 392 external_mini_cluster.cc:1428] Setting key 73c7e8dd4243adb1fce226eabe72678a
Jun 20 12:19:52 dist-test-slave-9b3m krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:19:52.076072 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.063126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33235 (local address 127.0.98.62:37337)
0620 12:19:52.063268 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.063271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.064081 (+ 810us) server_negotiation.cc:407] Connection header received
0620 12:19:52.065715 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.065718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.065767 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.065846 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:52.067241 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.067800 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.069148 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.069367 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.069454 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.072009 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:52.072015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:52.072021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:52.072055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:52.073864 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:52.074344 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:52.074348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:52.074352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:52.074439 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:52.074804 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:52.074807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:52.074808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:52.075052 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:52.075164 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.075727 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.075863 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:19:52.076848 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.063498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37337 (local address 127.0.98.2:33235)
0620 12:19:52.063938 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:52.063978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:52.065426 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:52.066050 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:52.066058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:52.066543 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:52.067091 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.067112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.067933 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.067936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.069004 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.069017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.069527 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.069531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.069745 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.070419 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:52.070435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:52.071837 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:52.073980 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:52.073984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:52.073995 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:52.074239 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:52.074547 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:52.074549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:52.074550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:52.074707 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:52.075184 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:52.075189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:52.075417 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 12:19:52.075629 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 12:19:52.075971 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260620 12:19:52.078085 4802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:19:52.078356 4802 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:52.078866 4802 heartbeater.cc:511] Master 127.0.98.62:37337 requested a full tablet report, sending...
I20260620 12:19:52.080025 4479 ts_manager.cc:194] Registered new tserver with Master: 0a4235a825354fa9a1310d78d00d8456 (127.0.98.2:46243)
I20260620 12:19:52.080859 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:19:52.094112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37337
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:52.208696 4807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:52.208926 4807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:52.208967 4807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:52.212905 4807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:52.212960 4807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:52.213086 4807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:52.218175 4807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:52.219045 4807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:52.220009 4807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:52.228837 4815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:52.228910 4812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:52.228981 4807 server_base.cc:1034] running on GCE node
W20260620 12:19:52.228834 4813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:52.229416 4807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:52.230067 4807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:52.231230 4807 hybrid_clock.cc:648] HybridClock initialized: now 1781957992231196 us; error 46 us; skew 500 ppm
Jun 20 12:19:52 dist-test-slave-9b3m krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:52.234005 4807 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:52.234704 4807 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:52.234794 4807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:52.235013 4807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:52.236928 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "b50d7933e1b64887b9a69e2f8dd8978e"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "462cab6c7df1199d3a2b3be581fe58c8"
server_key_iv: "05e4bce02f11ddce96d172b3bcd1d5f4"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.237540 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b50d7933e1b64887b9a69e2f8dd8978e"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "462cab6c7df1199d3a2b3be581fe58c8"
server_key_iv: "05e4bce02f11ddce96d172b3bcd1d5f4"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.240937 4807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:19:52.243263 4821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:52.244463 4807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:52.244601 4807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "b50d7933e1b64887b9a69e2f8dd8978e"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "462cab6c7df1199d3a2b3be581fe58c8"
server_key_iv: "05e4bce02f11ddce96d172b3bcd1d5f4"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.244735 4807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:52.265296 4807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:52.268502 4807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:52.268702 4807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:52.269347 4807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:52.270139 4807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:52.270210 4807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:52.270275 4807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:52.270324 4807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:52.279331 4807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37907
I20260620 12:19:52.279508 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37907 every 8 connection(s)
I20260620 12:19:52.279927 4807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:52.280330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 4807
I20260620 12:19:52.280548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:52.280900 392 external_mini_cluster.cc:1428] Setting key 6c06814657db33b7100111cfabd472e2
Jun 20 12:19:52 dist-test-slave-9b3m krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:52.293917 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.281900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39383 (local address 127.0.98.62:37337)
0620 12:19:52.282056 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.282059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.282970 (+ 911us) server_negotiation.cc:407] Connection header received
0620 12:19:52.284630 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.284633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.284686 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.284763 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:52.285935 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.286467 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.287574 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.287805 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.287915 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.290416 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:52.290419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:52.290421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:52.290445 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:52.292067 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:52.292505 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:52.292508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:52.292509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:52.292582 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:52.292894 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:52.292897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:52.292898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:52.293173 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:52.293267 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.293697 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.293796 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:19:52.294682 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.282329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37337 (local address 127.0.98.3:39383)
0620 12:19:52.282832 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:52.282873 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:52.284415 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:52.284932 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:52.284939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:52.285342 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:52.285796 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.285809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.286589 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.286593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.287447 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.287456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.287946 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.287950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.288119 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.288694 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:52.288709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:52.290249 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:52.292167 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:52.292171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:52.292181 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:52.292418 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:52.292662 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:52.292664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:52.292666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:52.292783 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:52.293272 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:52.293277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:52.293486 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 12:19:52.293633 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 12:19:52.293852 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":123,"threads_started":1}
I20260620 12:19:52.296041 4935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37337
I20260620 12:19:52.296305 4935 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:52.296854 4935 heartbeater.cc:511] Master 127.0.98.62:37337 requested a full tablet report, sending...
I20260620 12:19:52.297896 4478 ts_manager.cc:194] Registered new tserver with Master: b50d7933e1b64887b9a69e2f8dd8978e (127.0.98.3:37907)
I20260620 12:19:52.298465 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39383
I20260620 12:19:52.308295 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:19:52.318653 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.310262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.0.98.62:37337)
0620 12:19:52.310431 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.310438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.310600 (+ 162us) server_negotiation.cc:407] Connection header received
0620 12:19:52.310762 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.310765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.310816 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.310908 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:52.311700 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.312284 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.313273 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.313577 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.313696 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.315156 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:52.315166 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:52.315169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:52.315191 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:52.316708 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:52.317357 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:52.317362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:52.317364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:52.317458 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:52.317735 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:52.317737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:52.317739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:52.317953 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:52.318114 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.318376 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.318482 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 12:19:52.321688 4478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:19:52.324855 4478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:52.339597 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.332942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46243 (local address 127.0.0.1:47884)
0620 12:19:52.334307 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:52.334363 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:52.334507 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:52.335096 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:52.335101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:52.335119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:52.335238 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.335246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.336557 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.336560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.337847 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.337857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.338957 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.338962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.339136 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.339164 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:19:52.339248 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 12:19:52.339385 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":129,"spinlock_wait_cycles":1337472,"thread_start_us":135,"threads_started":1}
I20260620 12:19:52.339619 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.332898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34963 (local address 127.0.0.1:34842)
0620 12:19:52.334307 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:52.334363 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:52.334512 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:52.335253 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:52.335256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:52.335274 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:52.335368 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.335374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.336752 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.336755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.338152 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.338163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.339282 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.339286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.339368 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.339388 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:19:52.339444 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 12:19:52.339501 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":111,"thread_start_us":64,"threads_started":1}
I20260620 12:19:52.339740 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.333028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.0.98.1:34963)
0620 12:19:52.333876 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.333884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.334452 (+ 568us) server_negotiation.cc:407] Connection header received
0620 12:19:52.334705 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.334711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.334916 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.335051 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:52.335494 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.336603 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.338289 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.339146 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.339263 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.339407 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.339470 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.339563 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":94,"threads_started":1}
I20260620 12:19:52.339629 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.333743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.0.98.3:37907)
0620 12:19:52.334126 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.334142 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.334452 (+ 310us) server_negotiation.cc:407] Connection header received
0620 12:19:52.334654 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.334662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.334826 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.334983 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:52.335266 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.336456 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.338297 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.338808 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.338960 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.339111 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.339362 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.339468 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":154,"threads_started":1}
I20260620 12:19:52.340597 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.333822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.0.98.2:46243)
0620 12:19:52.334231 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.334249 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.334658 (+ 409us) server_negotiation.cc:407] Connection header received
0620 12:19:52.334737 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.334742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.334916 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.335384 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:52.335568 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.336407 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.337981 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.338802 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.340125 (+ 1323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.340288 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.340348 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.340437 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":116,"threads_started":1}
I20260620 12:19:52.340938 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.333218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37907 (local address 127.0.0.1:34768)
0620 12:19:52.334347 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:52.334362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:52.334513 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:52.334974 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:52.334980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:52.335001 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:52.335156 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.335162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.336599 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.336602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.338152 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.338163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.338958 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.338961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.339106 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.339150 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 12:19:52.339277 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 12:19:52.340797 (+ 1520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":1548,"spinlock_wait_cycles":4992,"thread_start_us":92,"threads_started":1}
I20260620 12:19:52.342967 4603 tablet_service.cc:1467] Processing CreateTablet for tablet 862c9ccbda89460086f0371e6bb1350d (DEFAULT_TABLE table=test-table [id=387068671adb4780bb29bf95d3d8bf40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:52.343320 4736 tablet_service.cc:1467] Processing CreateTablet for tablet 862c9ccbda89460086f0371e6bb1350d (DEFAULT_TABLE table=test-table [id=387068671adb4780bb29bf95d3d8bf40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:52.344162 4603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862c9ccbda89460086f0371e6bb1350d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:52.344333 4736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862c9ccbda89460086f0371e6bb1350d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:52.344087 4869 tablet_service.cc:1467] Processing CreateTablet for tablet 862c9ccbda89460086f0371e6bb1350d (DEFAULT_TABLE table=test-table [id=387068671adb4780bb29bf95d3d8bf40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:52.344974 4869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862c9ccbda89460086f0371e6bb1350d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:52.350822 4952 tablet_bootstrap.cc:492] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110: Bootstrap starting.
I20260620 12:19:52.352047 4954 tablet_bootstrap.cc:492] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Bootstrap starting.
I20260620 12:19:52.353098 4952 tablet_bootstrap.cc:654] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:52.353369 4953 tablet_bootstrap.cc:492] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Bootstrap starting.
I20260620 12:19:52.353883 4954 tablet_bootstrap.cc:654] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:52.354441 4952 log.cc:824] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:52.355060 4954 log.cc:824] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:52.355407 4953 tablet_bootstrap.cc:654] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:52.356453 4952 tablet_bootstrap.cc:492] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110: No bootstrap required, opened a new log
I20260620 12:19:52.356530 4953 log.cc:824] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:52.356686 4952 ts_tablet_manager.cc:1397] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:19:52.357033 4954 tablet_bootstrap.cc:492] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: No bootstrap required, opened a new log
I20260620 12:19:52.357285 4954 ts_tablet_manager.cc:1397] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260620 12:19:52.358485 4953 tablet_bootstrap.cc:492] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: No bootstrap required, opened a new log
I20260620 12:19:52.358695 4953 ts_tablet_manager.cc:1397] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:19:52.359787 4952 raft_consensus.cc:348] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.360034 4952 raft_consensus.cc:374] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:52.360101 4952 raft_consensus.cc:729] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a49b238654459ab647dbb11232e110, State: Initialized, Role: FOLLOWER
I20260620 12:19:52.360571 4952 consensus_queue.cc:260] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.361012 4954 raft_consensus.cc:348] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.361249 4669 heartbeater.cc:503] Master 127.0.98.62:37337 was elected leader, sending a full tablet report...
I20260620 12:19:52.361292 4954 raft_consensus.cc:374] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:52.361351 4954 raft_consensus.cc:729] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50d7933e1b64887b9a69e2f8dd8978e, State: Initialized, Role: FOLLOWER
I20260620 12:19:52.361775 4954 consensus_queue.cc:260] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.362251 4952 ts_tablet_manager.cc:1428] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:19:52.362654 4954 ts_tablet_manager.cc:1428] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 12:19:52.362993 4935 heartbeater.cc:503] Master 127.0.98.62:37337 was elected leader, sending a full tablet report...
I20260620 12:19:52.363437 4953 raft_consensus.cc:348] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.363710 4953 raft_consensus.cc:374] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:52.363775 4953 raft_consensus.cc:729] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4235a825354fa9a1310d78d00d8456, State: Initialized, Role: FOLLOWER
I20260620 12:19:52.364215 4953 consensus_queue.cc:260] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.364851 4802 heartbeater.cc:503] Master 127.0.98.62:37337 was elected leader, sending a full tablet report...
I20260620 12:19:52.365163 4953 ts_tablet_manager.cc:1428] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 12:19:52.530896 4936 tablet.cc:2369] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:19:52.562345 4803 tablet.cc:2369] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:19:52.596607 4670 tablet.cc:2369] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:19:52.612047 4960 raft_consensus.cc:482] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:52.612293 4960 raft_consensus.cc:504] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.613547 4960 leader_election.cc:288] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a49b238654459ab647dbb11232e110 (127.0.98.1:34963), b50d7933e1b64887b9a69e2f8dd8978e (127.0.98.3:37907)
I20260620 12:19:52.617601 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.613954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34799 (local address 127.0.98.1:34963)
0620 12:19:52.614098 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.614103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.614253 (+ 150us) server_negotiation.cc:407] Connection header received
0620 12:19:52.614435 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.614439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.614485 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.614550 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:52.614886 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.615385 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.616566 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.616955 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.617155 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.617222 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.617404 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.617471 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:19:52.617614 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.613954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39475 (local address 127.0.98.3:37907)
0620 12:19:52.614146 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.614151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.614334 (+ 183us) server_negotiation.cc:407] Connection header received
0620 12:19:52.614491 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.614494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.614549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.614629 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:19:52.614923 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.615398 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.616559 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.616989 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.617164 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.617226 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.617423 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.617490 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 12:19:52.617636 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.613842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37907 (local address 127.0.98.2:39475)
0620 12:19:52.614240 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:52.614254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:52.614380 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:52.614648 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:52.614651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:52.614689 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:52.614816 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.614821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.615512 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.615515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.616430 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.616439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.617168 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.617171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.617281 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.617303 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:19:52.617405 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 12:19:52.617474 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":61,"spinlock_wait_cycles":1664,"thread_start_us":47,"threads_started":1}
I20260620 12:19:52.618052 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.613842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34963 (local address 127.0.98.2:34799)
0620 12:19:52.614153 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:52.614175 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:52.614308 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:52.614615 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:52.614618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:52.614635 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:19:52.614772 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.614777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.615491 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.615497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.616423 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:52.616431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.617156 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.617160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:52.617257 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.617281 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:19:52.617387 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 12:19:52.617468 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":45,"spinlock_wait_cycles":15616,"thread_start_us":71,"threads_started":1}
I20260620 12:19:52.618263 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862c9ccbda89460086f0371e6bb1350d" candidate_uuid: "0a4235a825354fa9a1310d78d00d8456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" is_pre_election: true
I20260620 12:19:52.618566 4889 raft_consensus.cc:2454] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a4235a825354fa9a1310d78d00d8456 in term 0.
I20260620 12:19:52.618710 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862c9ccbda89460086f0371e6bb1350d" candidate_uuid: "0a4235a825354fa9a1310d78d00d8456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a49b238654459ab647dbb11232e110" is_pre_election: true
I20260620 12:19:52.618992 4623 raft_consensus.cc:2454] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a4235a825354fa9a1310d78d00d8456 in term 0.
I20260620 12:19:52.619079 4692 leader_election.cc:302] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4235a825354fa9a1310d78d00d8456, b50d7933e1b64887b9a69e2f8dd8978e; no voters:
I20260620 12:19:52.619328 4960 raft_consensus.cc:2781] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:52.619424 4960 raft_consensus.cc:482] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:52.619503 4960 raft_consensus.cc:3037] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:52.620580 4960 raft_consensus.cc:504] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.621135 4960 leader_election.cc:288] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [CANDIDATE]: Term 1 election: Requested vote from peers 55a49b238654459ab647dbb11232e110 (127.0.98.1:34963), b50d7933e1b64887b9a69e2f8dd8978e (127.0.98.3:37907)
I20260620 12:19:52.621567 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862c9ccbda89460086f0371e6bb1350d" candidate_uuid: "0a4235a825354fa9a1310d78d00d8456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b50d7933e1b64887b9a69e2f8dd8978e"
I20260620 12:19:52.621690 4889 raft_consensus.cc:3037] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:52.622115 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862c9ccbda89460086f0371e6bb1350d" candidate_uuid: "0a4235a825354fa9a1310d78d00d8456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a49b238654459ab647dbb11232e110"
I20260620 12:19:52.622217 4623 raft_consensus.cc:3037] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:52.623019 4889 raft_consensus.cc:2454] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a4235a825354fa9a1310d78d00d8456 in term 1.
I20260620 12:19:52.623339 4692 leader_election.cc:302] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4235a825354fa9a1310d78d00d8456, b50d7933e1b64887b9a69e2f8dd8978e; no voters:
I20260620 12:19:52.623534 4623 raft_consensus.cc:2454] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a4235a825354fa9a1310d78d00d8456 in term 1.
I20260620 12:19:52.623626 4960 raft_consensus.cc:2781] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:52.623837 4960 raft_consensus.cc:686] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 1 LEADER]: Becoming Leader. State: Replica: 0a4235a825354fa9a1310d78d00d8456, State: Running, Role: LEADER
I20260620 12:19:52.624153 4960 consensus_queue.cc:237] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } }
I20260620 12:19:52.626987 4479 catalog_manager.cc:5495] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a4235a825354fa9a1310d78d00d8456 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0a4235a825354fa9a1310d78d00d8456" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a4235a825354fa9a1310d78d00d8456" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 } health_report { overall_health: UNKNOWN } } }
I20260620 12:19:52.701018 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.697238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.0.98.2:46243)
0620 12:19:52.697386 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.697391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.697454 (+ 63us) server_negotiation.cc:407] Connection header received
0620 12:19:52.697679 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.697683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.697752 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.697926 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:19:52.698198 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.698712 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.699567 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:52.699791 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:52.699899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:52.700238 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:19:52.700651 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:19:52.700786 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.700825 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.700875 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:19:52.706640 4889 raft_consensus.cc:1261] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 1 FOLLOWER]: Refusing update from remote peer 0a4235a825354fa9a1310d78d00d8456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:52.706691 4623 raft_consensus.cc:1261] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [term 1 FOLLOWER]: Refusing update from remote peer 0a4235a825354fa9a1310d78d00d8456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:52.707317 4960 consensus_queue.cc:1035] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b50d7933e1b64887b9a69e2f8dd8978e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:52.707515 4963 consensus_queue.cc:1035] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a49b238654459ab647dbb11232e110" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:52.716387 4968 mvcc.cc:205] Tried to move back new op lower bound from 7298899938119716864 to 7298899937790861312. Current Snapshot: MvccSnapshot[applied={T|T < 7298899938119716864}]
I20260620 12:19:52.716549 4967 mvcc.cc:205] Tried to move back new op lower bound from 7298899938119716864 to 7298899937790861312. Current Snapshot: MvccSnapshot[applied={T|T < 7298899938119716864}]
I20260620 12:19:52.721875 4479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:52.722034 4479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:52.724284 4479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c13a1c601a54ecebfc6d3c4f2093e83: Sending DeleteTablet for 3 replicas of tablet 862c9ccbda89460086f0371e6bb1350d
I20260620 12:19:52.725026 4603 tablet_service.cc:1514] Processing DeleteTablet for tablet 862c9ccbda89460086f0371e6bb1350d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34842
I20260620 12:19:52.725128 4869 tablet_service.cc:1514] Processing DeleteTablet for tablet 862c9ccbda89460086f0371e6bb1350d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34768
I20260620 12:19:52.725294 4736 tablet_service.cc:1514] Processing DeleteTablet for tablet 862c9ccbda89460086f0371e6bb1350d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47884
I20260620 12:19:52.725539 4976 tablet_replica.cc:330] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110: stopping tablet replica
I20260620 12:19:52.725752 4977 tablet_replica.cc:330] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: stopping tablet replica
I20260620 12:19:52.725946 4976 raft_consensus.cc:2229] T 862c9ccbda89460086f0371e6bb1350d P 55a49b238654459ab647dbb11232e110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:52.725975 4977 raft_consensus.cc:2229] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:19:52.726219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 4541
I20260620 12:19:52.726256 4977 raft_consensus.cc:2258] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:52.727893 4975 tablet_replica.cc:330] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: stopping tablet replica
I20260620 12:19:52.728252 4975 raft_consensus.cc:2229] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:52.729691 4975 raft_consensus.cc:2258] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:52.729744 4977 ts_tablet_manager.cc:1905] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:52.731148 4975 ts_tablet_manager.cc:1905] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:52.732700 4977 ts_tablet_manager.cc:1918] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:19:52.732802 4977 log.cc:1196] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/862c9ccbda89460086f0371e6bb1350d
I20260620 12:19:52.733156 4977 ts_tablet_manager.cc:1939] T 862c9ccbda89460086f0371e6bb1350d P 0a4235a825354fa9a1310d78d00d8456: Deleting consensus metadata
I20260620 12:19:52.733953 4975 ts_tablet_manager.cc:1918] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 12:19:52.733954 4464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34963 (error 108)
I20260620 12:19:52.734126 4975 log.cc:1196] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/862c9ccbda89460086f0371e6bb1350d
I20260620 12:19:52.734478 4463 catalog_manager.cc:4841] TS 0a4235a825354fa9a1310d78d00d8456 (127.0.98.2:46243): tablet 862c9ccbda89460086f0371e6bb1350d (table test-table [id=387068671adb4780bb29bf95d3d8bf40]) successfully deleted
I20260620 12:19:52.734562 4975 ts_tablet_manager.cc:1939] T 862c9ccbda89460086f0371e6bb1350d P b50d7933e1b64887b9a69e2f8dd8978e: Deleting consensus metadata
I20260620 12:19:52.734870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 4674
I20260620 12:19:52.736140 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.735527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34963 (local address 127.0.0.1:34846)
0620 12:19:52.735950 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:52.736041 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318}
W20260620 12:19:52.736611 4464 catalog_manager.cc:4584] TS 55a49b238654459ab647dbb11232e110 (127.0.98.1:34963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 862c9ccbda89460086f0371e6bb1350d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34963: connect: Connection refused (error 111)
I20260620 12:19:52.737419 4466 catalog_manager.cc:4841] TS b50d7933e1b64887b9a69e2f8dd8978e (127.0.98.3:37907): tablet 862c9ccbda89460086f0371e6bb1350d (table test-table [id=387068671adb4780bb29bf95d3d8bf40]) successfully deleted
I20260620 12:19:52.740630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 4807
I20260620 12:19:52.746472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 4448
2026-06-20T12:19:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2518 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T12:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:52Z Disabled control of system clock
I20260620 12:19:52.776046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40967
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 12:19:52.886653 4982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:52.886906 4982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:52.886941 4982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:52.890940 4982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:52.890996 4982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:52.891012 4982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:52.891031 4982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:52.891045 4982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:52.895766 4982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:52.896920 4982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:52.898340 4982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:52.906616 4988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:52.906661 4987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:52.906791 4990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:52.906869 4982 server_base.cc:1034] running on GCE node
I20260620 12:19:52.907501 4982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:52.908561 4982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:52.909754 4982 hybrid_clock.cc:648] HybridClock initialized: now 1781957992909719 us; error 54 us; skew 500 ppm
I20260620 12:19:52.911584 4982 webserver.cc:460] Webserver started at http://127.0.98.62:42957/ using document root <none> and password file <none>
I20260620 12:19:52.912164 4982 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:52.912261 4982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:52.912489 4982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:52.914533 4982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "2b1ac25e22dc403ea4e25e67896c95cd"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "8841882fb11853a7e2c8a1b29db4440d"
server_key_iv: "aa4236b8bc7b70e43c68ba4c332091b2"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.915093 4982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b1ac25e22dc403ea4e25e67896c95cd"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "8841882fb11853a7e2c8a1b29db4440d"
server_key_iv: "aa4236b8bc7b70e43c68ba4c332091b2"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.918798 4982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:52.921356 4996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:52.922572 4982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:19:52.922717 4982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "2b1ac25e22dc403ea4e25e67896c95cd"
format_stamp: "Formatted at 2026-06-20 12:19:52 on dist-test-slave-9b3m"
server_key: "8841882fb11853a7e2c8a1b29db4440d"
server_key_iv: "aa4236b8bc7b70e43c68ba4c332091b2"
server_key_version: "encryptionkey@0"
I20260620 12:19:52.922828 4982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:52.927275 4982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:52.927865 4982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:52.928049 4982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:52.934535 4982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40967
I20260620 12:19:52.934576 5048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40967 every 8 connection(s)
I20260620 12:19:52.935431 4982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:52.938431 5049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:52.941407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 4982
I20260620 12:19:52.941535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:52.941828 392 external_mini_cluster.cc:1428] Setting key a26ba2059b32798dc8e28b98b79e6e27
I20260620 12:19:52.944686 5049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd: Bootstrap starting.
I20260620 12:19:52.947292 5049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:52.947414 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:52.943298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.0.98.62:40967)
0620 12:19:52.943904 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:52.943919 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:52.943954 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:19:52.944547 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:52.944565 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:52.944947 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:52.945318 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:52.946132 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:52.946136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:52.946147 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:52.946156 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:52.946208 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:52.946222 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:52.946328 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:52.946433 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 12:19:52.946729 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":126,"threads_started":1}
I20260620 12:19:52.948225 5049 log.cc:824] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:52.950255 5049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd: No bootstrap required, opened a new log
I20260620 12:19:52.953715 5049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1ac25e22dc403ea4e25e67896c95cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40967 } }
I20260620 12:19:52.953920 5049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:52.953960 5049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1ac25e22dc403ea4e25e67896c95cd, State: Initialized, Role: FOLLOWER
I20260620 12:19:52.954294 5049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1ac25e22dc403ea4e25e67896c95cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40967 } }
I20260620 12:19:52.954407 5049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:52.954444 5049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:52.954519 5049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:52.955315 5049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1ac25e22dc403ea4e25e67896c95cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40967 } }
I20260620 12:19:52.955619 5049 leader_election.cc:302] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b1ac25e22dc403ea4e25e67896c95cd; no voters:
I20260620 12:19:52.955909 5049 leader_election.cc:288] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:52.956064 5054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:52.956316 5054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [term 1 LEADER]: Becoming Leader. State: Replica: 2b1ac25e22dc403ea4e25e67896c95cd, State: Running, Role: LEADER
I20260620 12:19:52.956605 5054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1ac25e22dc403ea4e25e67896c95cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40967 } }
I20260620 12:19:52.957255 5049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:52.958421 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b1ac25e22dc403ea4e25e67896c95cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1ac25e22dc403ea4e25e67896c95cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40967 } } }
I20260620 12:19:52.958448 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b1ac25e22dc403ea4e25e67896c95cd. Latest consensus state: current_term: 1 leader_uuid: "2b1ac25e22dc403ea4e25e67896c95cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1ac25e22dc403ea4e25e67896c95cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40967 } } }
I20260620 12:19:52.958540 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:52.958559 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:52.958954 5063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:52.961831 5063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:52.966866 5063 catalog_manager.cc:1347] Generated new cluster ID: 8f6e316456c14a31ba10f89eb3d3a1be
I20260620 12:19:52.966936 5063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:52.990962 5063 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:52.991293 5063 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:53.000535 5063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd: Generated new TSK 0
I20260620 12:19:53.001410 5063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:19:53.012635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40967
--builtin_ntp_servers=127.0.98.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 12:19:53.120784 5073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:53.121011 5073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:53.121062 5073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:53.125869 5073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:53.125931 5073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:53.126005 5073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:53.133533 5073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:53.135167 5073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:53.136961 5073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:53.149082 5078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:53.149322 5073 server_base.cc:1034] running on GCE node
W20260620 12:19:53.149078 5079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:53.149192 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:53.150132 5073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:53.150904 5073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:53.152089 5073 hybrid_clock.cc:648] HybridClock initialized: now 1781957993152054 us; error 51 us; skew 500 ppm
I20260620 12:19:53.154136 5073 webserver.cc:460] Webserver started at http://127.0.98.1:41449/ using document root <none> and password file <none>
I20260620 12:19:53.154856 5073 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:53.154949 5073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:53.155223 5073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:53.157990 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "17c16fb7e69b44c3911100bf6c1a1863"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "0f98fade8e99c037332ba73b0ed1dd3e"
server_key_iv: "afd3d570e92bc803381cd84bc8680e3d"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.158635 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "17c16fb7e69b44c3911100bf6c1a1863"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "0f98fade8e99c037332ba73b0ed1dd3e"
server_key_iv: "afd3d570e92bc803381cd84bc8680e3d"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.163947 5073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:19:53.166543 5087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.167867 5073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:53.168015 5073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "17c16fb7e69b44c3911100bf6c1a1863"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "0f98fade8e99c037332ba73b0ed1dd3e"
server_key_iv: "afd3d570e92bc803381cd84bc8680e3d"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.168125 5073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:53.174021 5073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:53.174954 5073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:53.175132 5073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:53.175660 5073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:53.176544 5073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:53.176612 5073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.176672 5073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:53.176721 5073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.185163 5073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38151
I20260620 12:19:53.185384 5200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38151 every 8 connection(s)
I20260620 12:19:53.186053 5073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:53.188966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 5073
I20260620 12:19:53.189108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:53.189431 392 external_mini_cluster.cc:1428] Setting key 25b2d0f4a4b3ea1d19018d1124fbf714
I20260620 12:19:53.191669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40967
--builtin_ntp_servers=127.0.98.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 12:19:53.192559 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.187938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35017 (local address 127.0.98.62:40967)
0620 12:19:53.188066 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.188069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.189311 (+ 1242us) server_negotiation.cc:407] Connection header received
0620 12:19:53.190744 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.190748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.190787 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.190865 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.192098 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.192101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.192104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.192133 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.192151 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.192160 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.192253 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.192392 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.192433 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:19:53.193678 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.188314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40967 (local address 127.0.98.1:35017)
0620 12:19:53.189124 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.189178 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:53.190538 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:53.191014 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:53.191022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:53.191105 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:53.191757 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:19:53.191759 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:53.191796 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:19:53.191898 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:53.192241 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:53.192249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:53.192255 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:19:53.192433 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 12:19:53.192665 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":139,"threads_started":1}
I20260620 12:19:53.194926 5201 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40967
I20260620 12:19:53.195178 5201 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:53.195676 5201 heartbeater.cc:511] Master 127.0.98.62:40967 requested a full tablet report, sending...
I20260620 12:19:53.197494 5013 ts_manager.cc:194] Registered new tserver with Master: 17c16fb7e69b44c3911100bf6c1a1863 (127.0.98.1:38151)
W20260620 12:19:53.306412 5204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:53.306649 5204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:53.306689 5204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:53.310635 5204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:53.310679 5204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:53.310775 5204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:53.315527 5204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:53.316538 5204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:53.317915 5204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:53.327273 5209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:53.327277 5212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:53.327364 5204 server_base.cc:1034] running on GCE node
W20260620 12:19:53.327314 5210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:53.327980 5204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:53.328570 5204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:53.329762 5204 hybrid_clock.cc:648] HybridClock initialized: now 1781957993329742 us; error 27 us; skew 500 ppm
I20260620 12:19:53.331712 5204 webserver.cc:460] Webserver started at http://127.0.98.2:37329/ using document root <none> and password file <none>
I20260620 12:19:53.332299 5204 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:53.332361 5204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:53.332608 5204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:53.334656 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "8343f5086af342c48f48fd93565c2cf5"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "1ceb370b7fd5be7d7f2f1a3cbdab8bf9"
server_key_iv: "d8c6b69eb93120523ae5d365a5b329ab"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.335220 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8343f5086af342c48f48fd93565c2cf5"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "1ceb370b7fd5be7d7f2f1a3cbdab8bf9"
server_key_iv: "d8c6b69eb93120523ae5d365a5b329ab"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.338837 5204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 12:19:53.341228 5218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.342423 5204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:53.342586 5204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "8343f5086af342c48f48fd93565c2cf5"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "1ceb370b7fd5be7d7f2f1a3cbdab8bf9"
server_key_iv: "d8c6b69eb93120523ae5d365a5b329ab"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.342702 5204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:53.347251 5204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:53.347818 5204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:53.347992 5204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:53.348512 5204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:53.349406 5204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:53.349473 5204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.349550 5204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:53.349586 5204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.358189 5204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41761
I20260620 12:19:53.358296 5331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41761 every 8 connection(s)
I20260620 12:19:53.359062 5204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:53.364934 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.360832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60311 (local address 127.0.98.62:40967)
0620 12:19:53.360969 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.360973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.362044 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 12:19:53.363314 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.363317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.363360 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.363447 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.364462 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.364466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.364469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.364480 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.364501 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.364517 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.364619 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.364761 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.364805 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:19:53.365736 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.361285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40967 (local address 127.0.98.2:60311)
0620 12:19:53.361895 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.361932 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:53.363114 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:53.363617 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:53.363623 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:53.363707 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:53.364221 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:19:53.364222 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:53.364259 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:19:53.364320 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:53.364602 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:53.364610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:53.364616 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:19:53.364791 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 12:19:53.365010 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260620 12:19:53.366758 5332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40967
I20260620 12:19:53.366950 5332 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:53.367278 5332 heartbeater.cc:511] Master 127.0.98.62:40967 requested a full tablet report, sending...
I20260620 12:19:53.368176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 5204
I20260620 12:19:53.368307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:53.368302 5013 ts_manager.cc:194] Registered new tserver with Master: 8343f5086af342c48f48fd93565c2cf5 (127.0.98.2:41761)
I20260620 12:19:53.368598 392 external_mini_cluster.cc:1428] Setting key 36c11d2155ff9457550530169781a1d3
I20260620 12:19:53.370541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40967
--builtin_ntp_servers=127.0.98.20:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 12:19:53.478302 5335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:53.478536 5335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:53.478574 5335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:53.482540 5335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:53.482586 5335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:53.482641 5335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:53.487229 5335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:53.488205 5335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:53.489588 5335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:53.499148 5340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:53.499142 5341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:53.499300 5335 server_base.cc:1034] running on GCE node
W20260620 12:19:53.499142 5343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:53.499768 5335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:53.500481 5335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:53.501645 5335 hybrid_clock.cc:648] HybridClock initialized: now 1781957993501624 us; error 31 us; skew 500 ppm
I20260620 12:19:53.503436 5335 webserver.cc:460] Webserver started at http://127.0.98.3:38665/ using document root <none> and password file <none>
I20260620 12:19:53.504004 5335 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:53.504067 5335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:53.504271 5335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:53.506312 5335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "5d4251c4b8c74a019a7b1f34dd86e491"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "2c22817d9397946ebebb79d6e4a2ac9b"
server_key_iv: "92f43c8f5e6c7f8f8ae010284df4bad7"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.506865 5335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5d4251c4b8c74a019a7b1f34dd86e491"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "2c22817d9397946ebebb79d6e4a2ac9b"
server_key_iv: "92f43c8f5e6c7f8f8ae010284df4bad7"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.510380 5335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:53.512709 5349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.514045 5335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:53.514173 5335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "5d4251c4b8c74a019a7b1f34dd86e491"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "2c22817d9397946ebebb79d6e4a2ac9b"
server_key_iv: "92f43c8f5e6c7f8f8ae010284df4bad7"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.514276 5335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:53.518616 5335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:53.519156 5335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:53.519338 5335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:53.519845 5335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:53.520706 5335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:53.520776 5335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.520841 5335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:53.520891 5335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.530331 5335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35347
I20260620 12:19:53.530357 5462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35347 every 8 connection(s)
I20260620 12:19:53.531149 5335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:53.536394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 5335
I20260620 12:19:53.536556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:53.536870 392 external_mini_cluster.cc:1428] Setting key 0608ab57b9bdbe44949153fcce8886b1
I20260620 12:19:53.537269 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.532879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34829 (local address 127.0.98.62:40967)
0620 12:19:53.533011 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.533020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.534108 (+ 1088us) server_negotiation.cc:407] Connection header received
0620 12:19:53.535309 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.535312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.535348 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.535431 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.536650 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.536654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.536660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.536675 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.536698 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.536706 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.536781 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.537006 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.537107 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:19:53.538172 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.533316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40967 (local address 127.0.98.3:34829)
0620 12:19:53.533961 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.533999 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:53.535115 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:53.535593 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:53.535600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:53.535683 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:53.536300 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:19:53.536302 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:53.536358 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:19:53.536449 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:53.536796 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:53.536806 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:53.536813 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:19:53.536990 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 12:19:53.537254 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260620 12:19:53.539323 5463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40967
I20260620 12:19:53.539530 5463 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:53.539884 5463 heartbeater.cc:511] Master 127.0.98.62:40967 requested a full tablet report, sending...
I20260620 12:19:53.540797 5013 ts_manager.cc:194] Registered new tserver with Master: 5d4251c4b8c74a019a7b1f34dd86e491 (127.0.98.3:35347)
I20260620 12:19:53.550616 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:19:53.554256 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.552626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.0.98.62:40967)
0620 12:19:53.552752 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.552756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.552933 (+ 177us) server_negotiation.cc:407] Connection header received
0620 12:19:53.553203 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.553206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.553248 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.553317 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.553970 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.553973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.553975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.553985 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.554004 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.554012 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.554088 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.554130 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.554160 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:19:53.556998 5013 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:19:53.560132 5013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:19:53.571671 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.569247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35347 (local address 127.0.0.1:57266)
0620 12:19:53.569668 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.569679 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:53.569733 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:53.570303 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:53.570306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:53.570321 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:53.570927 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:19:53.570930 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:53.570955 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:19:53.571046 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:53.571331 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:53.571338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:53.571343 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:19:53.571455 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 12:19:53.571524 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":82,"threads_started":1}
I20260620 12:19:53.571947 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.569625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.0.98.3:35347)
0620 12:19:53.569954 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.569964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.569984 (+ 20us) server_negotiation.cc:407] Connection header received
0620 12:19:53.570027 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.570031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.570168 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.570532 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.571160 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.571168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.571180 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.571189 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.571228 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.571253 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.571365 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.571481 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.571568 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":157,"threads_started":1}
I20260620 12:19:53.571949 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.568498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38151 (local address 127.0.0.1:51104)
0620 12:19:53.569566 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.569583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:53.569665 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:53.570849 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:53.570855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:53.570872 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:53.571235 (+ 363us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:19:53.571237 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:53.571248 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:19:53.571274 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:53.571674 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:53.571677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:53.571680 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:19:53.571765 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 12:19:53.571799 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":130,"threads_started":1}
I20260620 12:19:53.572392 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.568735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.0.98.1:38151)
0620 12:19:53.569574 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.569592 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.569771 (+ 179us) server_negotiation.cc:407] Connection header received
0620 12:19:53.569835 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.569842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.570054 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.570182 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.571406 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.571438 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.571451 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.571463 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.571503 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.571535 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.571631 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.572140 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.572210 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":124,"threads_started":1}
I20260620 12:19:53.572861 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.568475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41761 (local address 127.0.0.1:56636)
0620 12:19:53.569286 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.569332 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:53.569425 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:53.570186 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:53.570191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:53.570265 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:53.571169 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:19:53.571172 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:53.571183 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:19:53.571211 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:53.572578 (+ 1367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:53.572581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:53.572583 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 12:19:53.572675 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 12:19:53.572707 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":86,"threads_started":1}
I20260620 12:19:53.572929 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.569194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.0.98.2:41761)
0620 12:19:53.569792 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.569804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.569828 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:19:53.569878 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.569883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.570045 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.570188 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.572280 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.572285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.572295 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.572304 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.572339 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.572358 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.572505 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.572705 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.572767 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":94,"threads_started":1}
I20260620 12:19:53.575129 5135 tablet_service.cc:1467] Processing CreateTablet for tablet 46caafddecdf4ff192c18ce92bdb87d9 (DEFAULT_TABLE table=test-table [id=5e2549dff64d4533bb148eff46b87cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:53.575584 5266 tablet_service.cc:1467] Processing CreateTablet for tablet 46caafddecdf4ff192c18ce92bdb87d9 (DEFAULT_TABLE table=test-table [id=5e2549dff64d4533bb148eff46b87cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:53.576222 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46caafddecdf4ff192c18ce92bdb87d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:53.576552 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46caafddecdf4ff192c18ce92bdb87d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:53.576283 5397 tablet_service.cc:1467] Processing CreateTablet for tablet 46caafddecdf4ff192c18ce92bdb87d9 (DEFAULT_TABLE table=test-table [id=5e2549dff64d4533bb148eff46b87cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:19:53.577178 5397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46caafddecdf4ff192c18ce92bdb87d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:53.582126 5480 tablet_bootstrap.cc:492] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863: Bootstrap starting.
I20260620 12:19:53.583243 5481 tablet_bootstrap.cc:492] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Bootstrap starting.
I20260620 12:19:53.584126 5482 tablet_bootstrap.cc:492] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: Bootstrap starting.
I20260620 12:19:53.585034 5480 tablet_bootstrap.cc:654] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:53.585552 5481 tablet_bootstrap.cc:654] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:53.585856 5480 log.cc:824] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:53.586139 5482 tablet_bootstrap.cc:654] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:53.586328 5481 log.cc:824] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:53.586864 5482 log.cc:824] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:53.587845 5480 tablet_bootstrap.cc:492] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863: No bootstrap required, opened a new log
I20260620 12:19:53.588042 5480 ts_tablet_manager.cc:1397] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 12:19:53.588143 5481 tablet_bootstrap.cc:492] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: No bootstrap required, opened a new log
I20260620 12:19:53.588315 5481 ts_tablet_manager.cc:1397] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:19:53.588653 5482 tablet_bootstrap.cc:492] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: No bootstrap required, opened a new log
I20260620 12:19:53.588819 5482 ts_tablet_manager.cc:1397] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:19:53.592057 5480 raft_consensus.cc:348] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.592338 5480 raft_consensus.cc:374] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:53.592386 5480 raft_consensus.cc:729] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c16fb7e69b44c3911100bf6c1a1863, State: Initialized, Role: FOLLOWER
I20260620 12:19:53.592772 5481 raft_consensus.cc:348] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.593001 5481 raft_consensus.cc:374] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:53.593079 5481 raft_consensus.cc:729] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8343f5086af342c48f48fd93565c2cf5, State: Initialized, Role: FOLLOWER
I20260620 12:19:53.593081 5482 raft_consensus.cc:348] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.593304 5482 raft_consensus.cc:374] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:53.593348 5482 raft_consensus.cc:729] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4251c4b8c74a019a7b1f34dd86e491, State: Initialized, Role: FOLLOWER
I20260620 12:19:53.593529 5481 consensus_queue.cc:260] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.593710 5482 consensus_queue.cc:260] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.594280 5332 heartbeater.cc:503] Master 127.0.98.62:40967 was elected leader, sending a full tablet report...
I20260620 12:19:53.594328 5480 consensus_queue.cc:260] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.594628 5482 ts_tablet_manager.cc:1428] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:19:53.594923 5463 heartbeater.cc:503] Master 127.0.98.62:40967 was elected leader, sending a full tablet report...
I20260620 12:19:53.594993 5201 heartbeater.cc:503] Master 127.0.98.62:40967 was elected leader, sending a full tablet report...
I20260620 12:19:53.595284 5480 ts_tablet_manager.cc:1428] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:19:53.596307 5481 ts_tablet_manager.cc:1428] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
W20260620 12:19:53.609978 5333 tablet.cc:2369] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:19:53.654943 5486 raft_consensus.cc:482] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:19:53.655140 5486 raft_consensus.cc:504] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.656195 5486 leader_election.cc:288] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c16fb7e69b44c3911100bf6c1a1863 (127.0.98.1:38151), 5d4251c4b8c74a019a7b1f34dd86e491 (127.0.98.3:35347)
I20260620 12:19:53.658246 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.656426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35347 (local address 127.0.98.2:39011)
0620 12:19:53.656543 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.656558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:53.656640 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:53.656989 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:53.656991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:53.657025 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:53.657628 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:19:53.657630 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:53.657646 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:19:53.657669 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:53.657997 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:53.658001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:53.658004 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:19:53.658090 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:19:53.658126 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 12:19:53.658250 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.656606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39011 (local address 127.0.98.3:35347)
0620 12:19:53.656759 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.656764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.656781 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:19:53.656827 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.656830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.656869 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.656957 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.657810 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.657815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.657819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.657833 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.657860 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.657871 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.657998 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.658105 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.658145 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:19:53.658532 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.656554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56523 (local address 127.0.98.1:38151)
0620 12:19:53.656695 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.656700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.657071 (+ 371us) server_negotiation.cc:407] Connection header received
0620 12:19:53.657185 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.657190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.657237 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.657314 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.658128 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.658131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.658133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.658149 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.658166 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.658173 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.658258 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.658372 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.658419 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:19:53.658872 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.656426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38151 (local address 127.0.98.2:56523)
0620 12:19:53.656949 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.656962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:53.657086 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:53.657356 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:53.657361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:53.657378 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:53.657975 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:19:53.657976 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:53.657986 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:19:53.658019 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:53.658254 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:53.658257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:53.658260 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:19:53.658702 (+ 442us) client_negotiation.cc:240] Negotiation successful
0620 12:19:53.658735 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":102,"thread_start_us":127,"threads_started":1}
I20260620 12:19:53.658926 5417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46caafddecdf4ff192c18ce92bdb87d9" candidate_uuid: "8343f5086af342c48f48fd93565c2cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" is_pre_election: true
I20260620 12:19:53.659240 5417 raft_consensus.cc:2454] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8343f5086af342c48f48fd93565c2cf5 in term 0.
I20260620 12:19:53.659440 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46caafddecdf4ff192c18ce92bdb87d9" candidate_uuid: "8343f5086af342c48f48fd93565c2cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c16fb7e69b44c3911100bf6c1a1863" is_pre_election: true
I20260620 12:19:53.659695 5155 raft_consensus.cc:2454] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8343f5086af342c48f48fd93565c2cf5 in term 0.
I20260620 12:19:53.659734 5221 leader_election.cc:302] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4251c4b8c74a019a7b1f34dd86e491, 8343f5086af342c48f48fd93565c2cf5; no voters:
I20260620 12:19:53.659999 5486 raft_consensus.cc:2781] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:19:53.660089 5486 raft_consensus.cc:482] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:19:53.660131 5486 raft_consensus.cc:3037] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:53.661167 5486 raft_consensus.cc:504] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.661495 5486 leader_election.cc:288] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [CANDIDATE]: Term 1 election: Requested vote from peers 17c16fb7e69b44c3911100bf6c1a1863 (127.0.98.1:38151), 5d4251c4b8c74a019a7b1f34dd86e491 (127.0.98.3:35347)
I20260620 12:19:53.661785 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46caafddecdf4ff192c18ce92bdb87d9" candidate_uuid: "8343f5086af342c48f48fd93565c2cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c16fb7e69b44c3911100bf6c1a1863"
I20260620 12:19:53.661851 5417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46caafddecdf4ff192c18ce92bdb87d9" candidate_uuid: "8343f5086af342c48f48fd93565c2cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4251c4b8c74a019a7b1f34dd86e491"
I20260620 12:19:53.661917 5155 raft_consensus.cc:3037] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:53.661994 5417 raft_consensus.cc:3037] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:53.663051 5155 raft_consensus.cc:2454] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8343f5086af342c48f48fd93565c2cf5 in term 1.
I20260620 12:19:53.663277 5417 raft_consensus.cc:2454] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8343f5086af342c48f48fd93565c2cf5 in term 1.
I20260620 12:19:53.663333 5222 leader_election.cc:302] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c16fb7e69b44c3911100bf6c1a1863, 8343f5086af342c48f48fd93565c2cf5; no voters:
I20260620 12:19:53.663618 5486 raft_consensus.cc:2781] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:53.663846 5486 raft_consensus.cc:686] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 1 LEADER]: Becoming Leader. State: Replica: 8343f5086af342c48f48fd93565c2cf5, State: Running, Role: LEADER
I20260620 12:19:53.664170 5486 consensus_queue.cc:237] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } }
I20260620 12:19:53.667328 5013 catalog_manager.cc:5495] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8343f5086af342c48f48fd93565c2cf5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8343f5086af342c48f48fd93565c2cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343f5086af342c48f48fd93565c2cf5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 } health_report { overall_health: UNKNOWN } } }
W20260620 12:19:53.687214 5202 tablet.cc:2369] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:19:53.693181 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.691692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.98.2:41761)
0620 12:19:53.691822 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.691825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.691838 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:19:53.691977 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.691980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.692046 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.692131 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.692770 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.692773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.692775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.692785 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.692803 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.692811 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.692881 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.692935 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.692967 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:19:53.698805 5155 raft_consensus.cc:1261] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 1 FOLLOWER]: Refusing update from remote peer 8343f5086af342c48f48fd93565c2cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:53.698805 5417 raft_consensus.cc:1261] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 1 FOLLOWER]: Refusing update from remote peer 8343f5086af342c48f48fd93565c2cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:19:53.699837 5490 consensus_queue.cc:1035] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4251c4b8c74a019a7b1f34dd86e491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:53.700060 5486 consensus_queue.cc:1035] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c16fb7e69b44c3911100bf6c1a1863" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:53.706539 5493 mvcc.cc:205] Tried to move back new op lower bound from 7298899942182858752 to 7298899942051217408. Current Snapshot: MvccSnapshot[applied={T|T < 7298899942182858752}]
I20260620 12:19:53.708398 5494 mvcc.cc:205] Tried to move back new op lower bound from 7298899942182858752 to 7298899942051217408. Current Snapshot: MvccSnapshot[applied={T|T < 7298899942182858752}]
I20260620 12:19:53.716279 5013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:53.716457 5013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:19:53.718756 5013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b1ac25e22dc403ea4e25e67896c95cd: Sending DeleteTablet for 3 replicas of tablet 46caafddecdf4ff192c18ce92bdb87d9
I20260620 12:19:53.719560 5266 tablet_service.cc:1514] Processing DeleteTablet for tablet 46caafddecdf4ff192c18ce92bdb87d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:53 UTC) from {username='slave'} at 127.0.0.1:56636
I20260620 12:19:53.719560 5397 tablet_service.cc:1514] Processing DeleteTablet for tablet 46caafddecdf4ff192c18ce92bdb87d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:53 UTC) from {username='slave'} at 127.0.0.1:57266
I20260620 12:19:53.719547 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet 46caafddecdf4ff192c18ce92bdb87d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:19:53 UTC) from {username='slave'} at 127.0.0.1:51104
I20260620 12:19:53.720034 5504 tablet_replica.cc:330] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863: stopping tablet replica
I20260620 12:19:53.720213 5502 tablet_replica.cc:330] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: stopping tablet replica
I20260620 12:19:53.720345 5504 raft_consensus.cc:2229] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:53.720458 5502 raft_consensus.cc:2229] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:19:53.720650 5504 raft_consensus.cc:2258] T 46caafddecdf4ff192c18ce92bdb87d9 P 17c16fb7e69b44c3911100bf6c1a1863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:53.720837 5502 raft_consensus.cc:2258] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:53.720871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 5073
I20260620 12:19:53.721196 5503 tablet_replica.cc:330] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: stopping tablet replica
I20260620 12:19:53.721690 5503 raft_consensus.cc:2229] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:53.721959 5503 raft_consensus.cc:2258] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:53.723589 5503 ts_tablet_manager.cc:1905] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:53.724332 5502 ts_tablet_manager.cc:1905] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:53.726382 5503 ts_tablet_manager.cc:1918] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:19:53.726493 5503 log.cc:1196] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/46caafddecdf4ff192c18ce92bdb87d9
I20260620 12:19:53.726833 5503 ts_tablet_manager.cc:1939] T 46caafddecdf4ff192c18ce92bdb87d9 P 5d4251c4b8c74a019a7b1f34dd86e491: Deleting consensus metadata
I20260620 12:19:53.727504 5502 ts_tablet_manager.cc:1918] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:19:53.727651 5502 log.cc:1196] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/46caafddecdf4ff192c18ce92bdb87d9
I20260620 12:19:53.728006 5502 ts_tablet_manager.cc:1939] T 46caafddecdf4ff192c18ce92bdb87d9 P 8343f5086af342c48f48fd93565c2cf5: Deleting consensus metadata
I20260620 12:19:53.728246 4999 catalog_manager.cc:4841] TS 5d4251c4b8c74a019a7b1f34dd86e491 (127.0.98.3:35347): tablet 46caafddecdf4ff192c18ce92bdb87d9 (table test-table [id=5e2549dff64d4533bb148eff46b87cf5]) successfully deleted
I20260620 12:19:53.729137 4998 catalog_manager.cc:4841] TS 8343f5086af342c48f48fd93565c2cf5 (127.0.98.2:41761): tablet 46caafddecdf4ff192c18ce92bdb87d9 (table test-table [id=5e2549dff64d4533bb148eff46b87cf5]) successfully deleted
W20260620 12:19:53.729487 5000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38151 (error 108)
I20260620 12:19:53.730294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 5204
I20260620 12:19:53.730554 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.730191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38151 (local address 127.0.0.1:51110)
0620 12:19:53.730370 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:53.730447 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 12:19:53.731045 5000 catalog_manager.cc:4584] TS 17c16fb7e69b44c3911100bf6c1a1863 (127.0.98.1:38151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46caafddecdf4ff192c18ce92bdb87d9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38151: connect: Connection refused (error 111)
I20260620 12:19:53.736065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 5335
I20260620 12:19:53.741510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 4982
2026-06-20T12:19:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (993 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 12:19:53.762425 392 mini_oidc.cc:152] Starting JWKS server
I20260620 12:19:53.763902 392 webserver.cc:460] Webserver started at https://127.0.0.1:41495/ using document root <none> and password file <none>
I20260620 12:19:53.764034 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:19:53.764240 392 webserver.cc:460] Webserver started at http://127.0.0.1:40013/ using document root <none> and password file <none>
2026-06-20T12:19:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:53Z Disabled control of system clock
I20260620 12:19:53.772944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37767
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37767
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41495/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/testchainca.cert with env {}
W20260620 12:19:53.880849 5511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:53.881114 5511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:53.881155 5511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:53.885713 5511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:53.885782 5511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:53.885797 5511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:53.885813 5511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:53.885828 5511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:19:53.885844 5511 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:19:53.885857 5511 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41495/jwks.json
I20260620 12:19:53.891185 5511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41495/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:53.892207 5511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:53.893188 5511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:53.901298 5519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:53.901300 5517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:53.901376 5516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:53.901723 5511 server_base.cc:1034] running on GCE node
I20260620 12:19:53.902240 5511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:53.903281 5511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:53.904443 5511 hybrid_clock.cc:648] HybridClock initialized: now 1781957993904427 us; error 32 us; skew 500 ppm
I20260620 12:19:53.906313 5511 webserver.cc:460] Webserver started at http://127.0.98.62:44335/ using document root <none> and password file <none>
I20260620 12:19:53.906889 5511 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:53.906955 5511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:53.907171 5511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:53.909219 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "406a553bc7677e98e51797f9d5e715a6"
server_key_iv: "c5db2afaede3ff80ea5635930597f5c6"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.909787 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "406a553bc7677e98e51797f9d5e715a6"
server_key_iv: "c5db2afaede3ff80ea5635930597f5c6"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.913336 5511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:53.915741 5525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:53.916966 5511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:53.917132 5511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788"
format_stamp: "Formatted at 2026-06-20 12:19:53 on dist-test-slave-9b3m"
server_key: "406a553bc7677e98e51797f9d5e715a6"
server_key_iv: "c5db2afaede3ff80ea5635930597f5c6"
server_key_version: "encryptionkey@0"
I20260620 12:19:53.917253 5511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:53.932905 5511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:53.933583 5511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:53.933773 5511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:53.940140 5511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37767
I20260620 12:19:53.940145 5577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37767 every 8 connection(s)
I20260620 12:19:53.941016 5511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:53.944082 5578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:53.948920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 5511
I20260620 12:19:53.949060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:53.949244 5578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788: Bootstrap starting.
I20260620 12:19:53.949379 392 external_mini_cluster.cc:1428] Setting key 6a407f11ed4d54b2cf3dbdd3ffcd3f8c
I20260620 12:19:53.951619 5578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:53.952342 5578 log.cc:824] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:53.954085 5578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788: No bootstrap required, opened a new log
I20260620 12:19:53.958180 5578 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } }
I20260620 12:19:53.958401 5578 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:53.958465 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:53.950653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.98.62:37767)
0620 12:19:53.951264 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:53.951278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:53.951312 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:19:53.951968 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:53.951992 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:53.952379 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:53.952740 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:53.953207 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:53.954127 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:53.955359 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:53.955806 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:53.955921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:53.956698 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:53.956703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:53.956715 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:53.957151 (+ 436us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:53.957204 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:53.957223 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:53.957298 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:53.957475 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 12:19:53.957732 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
I20260620 12:19:53.958478 5578 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe9a99550a1c4f87b52d6d0dcf6a3788, State: Initialized, Role: FOLLOWER
I20260620 12:19:53.958997 5578 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } }
I20260620 12:19:53.959143 5578 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:53.959203 5578 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:53.959288 5578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:53.960108 5578 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } }
I20260620 12:19:53.960424 5578 leader_election.cc:302] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe9a99550a1c4f87b52d6d0dcf6a3788; no voters:
I20260620 12:19:53.960721 5578 leader_election.cc:288] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:53.960784 5583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:53.960965 5583 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [term 1 LEADER]: Becoming Leader. State: Replica: fe9a99550a1c4f87b52d6d0dcf6a3788, State: Running, Role: LEADER
I20260620 12:19:53.961237 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:19:53.961272 5583 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } }
I20260620 12:19:53.961910 5578 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:53.963567 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } } }
I20260620 12:19:53.963723 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:53.964079 5584 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe9a99550a1c4f87b52d6d0dcf6a3788. Latest consensus state: current_term: 1 leader_uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9a99550a1c4f87b52d6d0dcf6a3788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37767 } } }
I20260620 12:19:53.964187 5584 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788 [sys.catalog]: This master's current role is: LEADER
W20260620 12:19:53.966475 5599 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:19:53.966536 5599 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:19:53.966615 5598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:53.967227 5598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:53.972258 5598 catalog_manager.cc:1347] Generated new cluster ID: 43af33cd8380492d8df5d02fe3c54e29
I20260620 12:19:53.972319 5598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:53.988531 5598 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:53.989789 5598 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:54.009299 5598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe9a99550a1c4f87b52d6d0dcf6a3788: Generated new TSK 0
I20260620 12:19:54.010109 5598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:19:54.039867 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:54.021574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.98.62:37767)
0620 12:19:54.021700 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:54.021704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:54.021898 (+ 194us) server_negotiation.cc:407] Connection header received
0620 12:19:54.022077 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:54.022080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:54.022133 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:54.022238 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:19:54.022489 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.023312 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.024158 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.024501 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.024600 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.024908 (+ 308us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:19:54.039389 (+ 14481us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 12:19:54.039689 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 12:19:54.039963 5581 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
E20260620 12:19:54.040573 5608 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 12:19:54.040645 5608 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 12:19:54.041890 5607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:19:54.021493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37767 (local address 127.0.0.1:60560)
0620 12:19:54.021815 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:54.021832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:54.021934 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:54.022252 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:54.022254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:54.022260 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:19:54.022391 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:54.022396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.023430 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.023433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:54.024052 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:54.024058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.024604 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.024619 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:54.024687 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.024725 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:19:54.039659 (+ 14934us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 12:19:54.040005 (+ 346us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:37767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 12:19:54.042798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 5511
2026-06-20T12:19:54Z chronyd exiting
I20260620 12:19:54.081764 392 test_util.cc:175] -----------------------------------------------
I20260620 12:19:54.081872 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781957967886955-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 12:19:54.082943 392 mini_oidc.cc:152] Starting JWKS server
I20260620 12:19:54.083784 392 webserver.cc:460] Webserver started at https://127.0.0.1:45951/ using document root <none> and password file <none>
I20260620 12:19:54.083920 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:19:54.084185 392 webserver.cc:460] Webserver started at http://127.0.0.1:37635/ using document root <none> and password file <none>
2026-06-20T12:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:54Z Disabled control of system clock
I20260620 12:19:54.092553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39313
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39313
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45951/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/testchainca.cert with env {}
W20260620 12:19:54.196698 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:54.196933 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:54.196970 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:54.201171 5614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:54.201222 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:54.201237 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:54.201254 5614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:54.201308 5614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:19:54.201330 5614 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:19:54.201363 5614 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45951/jwks.json
I20260620 12:19:54.206431 5614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45951/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:54.207386 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:54.208345 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:54.216681 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:54.216763 5614 server_base.cc:1034] running on GCE node
W20260620 12:19:54.216732 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:54.216688 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:54.217521 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:54.218590 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:54.219770 5614 hybrid_clock.cc:648] HybridClock initialized: now 1781957994219736 us; error 48 us; skew 500 ppm
I20260620 12:19:54.221598 5614 webserver.cc:460] Webserver started at http://127.0.98.62:44703/ using document root <none> and password file <none>
I20260620 12:19:54.222157 5614 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:54.222246 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:54.222462 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:54.224382 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "8baa7e1efe2043e1bb455731b0939957"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "a8b84f0ed774f689b4265f0e7ce48c16"
server_key_iv: "43e5f248ea4bbca53582c098c9b841d4"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.224929 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "8baa7e1efe2043e1bb455731b0939957"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "a8b84f0ed774f689b4265f0e7ce48c16"
server_key_iv: "43e5f248ea4bbca53582c098c9b841d4"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.228511 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:19:54.230890 5628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:54.232116 5614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:54.232259 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "8baa7e1efe2043e1bb455731b0939957"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "a8b84f0ed774f689b4265f0e7ce48c16"
server_key_iv: "43e5f248ea4bbca53582c098c9b841d4"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.232374 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:54.257910 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:54.258595 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:54.258816 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:54.266122 5614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39313
I20260620 12:19:54.266130 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39313 every 8 connection(s)
I20260620 12:19:54.267074 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:54.267436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 5614
I20260620 12:19:54.267572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:54.267897 392 external_mini_cluster.cc:1428] Setting key 82926524fd5edca39e0c752456cea63c
I20260620 12:19:54.271220 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:54.276757 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:54.269203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.0.98.62:39313)
0620 12:19:54.269815 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:54.269833 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:54.269870 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:19:54.270498 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:54.270513 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:54.270895 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:54.271220 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:54.271638 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.272612 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.273596 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.274064 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.274241 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.275155 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:54.275160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:54.275171 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:54.275549 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:54.275601 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:54.275620 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:54.275729 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:54.275884 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 12:19:54.276118 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260620 12:19:54.277086 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957: Bootstrap starting.
I20260620 12:19:54.279675 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:19:54.279695 5681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:54.280453 5681 log.cc:824] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:54.282405 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957: No bootstrap required, opened a new log
I20260620 12:19:54.286139 5681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa7e1efe2043e1bb455731b0939957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39313 } }
I20260620 12:19:54.286458 5681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:54.286535 5681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8baa7e1efe2043e1bb455731b0939957, State: Initialized, Role: FOLLOWER
I20260620 12:19:54.286937 5681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa7e1efe2043e1bb455731b0939957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39313 } }
I20260620 12:19:54.287062 5681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:54.287104 5681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:54.287175 5681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:54.288086 5681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa7e1efe2043e1bb455731b0939957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39313 } }
I20260620 12:19:54.288379 5681 leader_election.cc:302] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8baa7e1efe2043e1bb455731b0939957; no voters:
I20260620 12:19:54.288632 5681 leader_election.cc:288] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:54.288789 5686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:54.289032 5686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [term 1 LEADER]: Becoming Leader. State: Replica: 8baa7e1efe2043e1bb455731b0939957, State: Running, Role: LEADER
I20260620 12:19:54.289413 5686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa7e1efe2043e1bb455731b0939957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39313 } }
I20260620 12:19:54.289866 5681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:54.290910 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8baa7e1efe2043e1bb455731b0939957. Latest consensus state: current_term: 1 leader_uuid: "8baa7e1efe2043e1bb455731b0939957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa7e1efe2043e1bb455731b0939957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39313 } } }
I20260620 12:19:54.291028 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:54.291342 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8baa7e1efe2043e1bb455731b0939957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa7e1efe2043e1bb455731b0939957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39313 } } }
I20260620 12:19:54.291426 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:54.291494 5695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:54.294270 5695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:54.299088 5695 catalog_manager.cc:1347] Generated new cluster ID: 28a58db6e31a4003b0a213f8dbd4de46
I20260620 12:19:54.299146 5695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:54.305426 5695 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:54.306334 5695 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:54.312896 5695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8baa7e1efe2043e1bb455731b0939957: Generated new TSK 0
I20260620 12:19:54.313513 5695 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 12:19:54.340752 5711 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 12:19:54.340847 5711 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 12:19:54.341564 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:54.325810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.0.98.62:39313)
0620 12:19:54.325957 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:54.325961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:54.326385 (+ 424us) server_negotiation.cc:407] Connection header received
0620 12:19:54.326585 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:54.326589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:54.326645 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:54.326761 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:19:54.327014 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.327529 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.328425 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.328643 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.328750 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.329087 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:19:54.341172 (+ 12085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 12:19:54.341420 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 12:19:54.341627 5684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 12:19:54.341656 5710 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:19:54.325810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39313 (local address 127.0.0.1:58666)
0620 12:19:54.326304 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:54.326321 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:54.326436 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:54.326773 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:54.326777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:54.326784 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:19:54.326908 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:54.326913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.327658 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.327663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:54.328300 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:54.328308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.328741 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.328759 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:54.328852 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.328880 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:19:54.341346 (+ 12466us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 12:19:54.341493 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39313: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":249,"threads_started":1}
I20260620 12:19:54.342340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 5614
2026-06-20T12:19:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (298 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 12:19:54.381383 392 mini_oidc.cc:152] Starting JWKS server
I20260620 12:19:54.382211 392 webserver.cc:460] Webserver started at https://127.0.0.1:34771/ using document root <none> and password file <none>
I20260620 12:19:54.382292 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 12:19:54.382493 392 webserver.cc:460] Webserver started at http://127.0.0.1:42319/ using document root <none> and password file <none>
2026-06-20T12:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:54Z Disabled control of system clock
I20260620 12:19:54.391350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41549
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 12:19:54.498169 5717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:54.498406 5717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:54.498442 5717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:54.502712 5717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:54.502774 5717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:54.502790 5717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:54.502806 5717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:54.502821 5717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:19:54.502836 5717 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:19:54.502849 5717 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34771/jwks.json
I20260620 12:19:54.507493 5717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:54.508419 5717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:54.509689 5717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:54.517688 5723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:54.517728 5725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:54.517900 5717 server_base.cc:1034] running on GCE node
W20260620 12:19:54.517731 5722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:54.518464 5717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:54.519554 5717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:54.520759 5717 hybrid_clock.cc:648] HybridClock initialized: now 1781957994520744 us; error 27 us; skew 500 ppm
I20260620 12:19:54.522691 5717 webserver.cc:460] Webserver started at http://127.0.98.62:35637/ using document root <none> and password file <none>
I20260620 12:19:54.523247 5717 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:54.523339 5717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:54.523579 5717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:54.525686 5717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "62b0559f88f54a8bbdc64f07d69c0191"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "221f8f68fc62f6d8dd9fbd5cc540ff84"
server_key_iv: "30ff831f2b1fa8e95e57098c6c63f335"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.526276 5717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "62b0559f88f54a8bbdc64f07d69c0191"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "221f8f68fc62f6d8dd9fbd5cc540ff84"
server_key_iv: "30ff831f2b1fa8e95e57098c6c63f335"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.529903 5717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:54.532316 5731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:54.533664 5717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:19:54.533818 5717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "62b0559f88f54a8bbdc64f07d69c0191"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "221f8f68fc62f6d8dd9fbd5cc540ff84"
server_key_iv: "30ff831f2b1fa8e95e57098c6c63f335"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.533936 5717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:54.554314 5717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:54.555030 5717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:54.555258 5717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:54.562685 5717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41549
I20260620 12:19:54.562704 5783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41549 every 8 connection(s)
I20260620 12:19:54.563675 5717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:54.566828 5784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:54.566818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 5717
I20260620 12:19:54.566977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:54.567281 392 external_mini_cluster.cc:1428] Setting key 0835a542d648dcf2f7b59776ef6ad5ae
I20260620 12:19:54.573927 5784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191: Bootstrap starting.
I20260620 12:19:54.576195 5784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:54.577039 5784 log.cc:824] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:54.577744 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:54.568687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.0.98.62:41549)
0620 12:19:54.569386 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:54.569405 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:54.569451 (+ 46us) server_negotiation.cc:407] Connection header received
0620 12:19:54.570231 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:54.570256 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:54.570763 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:54.571139 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:54.571565 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.573103 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.574133 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.574678 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.574872 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.575764 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:54.575769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:54.575781 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:54.576179 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:54.576237 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:54.576257 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:54.576392 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:54.576589 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 12:19:54.576894 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
I20260620 12:19:54.579049 5784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191: No bootstrap required, opened a new log
I20260620 12:19:54.580647 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:19:54.583256 5784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b0559f88f54a8bbdc64f07d69c0191" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260620 12:19:54.583465 5784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:54.583539 5784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b0559f88f54a8bbdc64f07d69c0191, State: Initialized, Role: FOLLOWER
I20260620 12:19:54.583956 5784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b0559f88f54a8bbdc64f07d69c0191" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260620 12:19:54.584074 5784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:54.584139 5784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:54.584254 5784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:54.585160 5784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b0559f88f54a8bbdc64f07d69c0191" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260620 12:19:54.585481 5784 leader_election.cc:302] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b0559f88f54a8bbdc64f07d69c0191; no voters:
I20260620 12:19:54.585783 5784 leader_election.cc:288] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:54.585930 5789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:54.586184 5789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [term 1 LEADER]: Becoming Leader. State: Replica: 62b0559f88f54a8bbdc64f07d69c0191, State: Running, Role: LEADER
I20260620 12:19:54.586516 5789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b0559f88f54a8bbdc64f07d69c0191" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260620 12:19:54.586990 5784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:54.588100 5791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b0559f88f54a8bbdc64f07d69c0191. Latest consensus state: current_term: 1 leader_uuid: "62b0559f88f54a8bbdc64f07d69c0191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b0559f88f54a8bbdc64f07d69c0191" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } } }
I20260620 12:19:54.588130 5790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b0559f88f54a8bbdc64f07d69c0191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b0559f88f54a8bbdc64f07d69c0191" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } } }
I20260620 12:19:54.588234 5790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:54.588233 5791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:54.588650 5798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:54.591676 5798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:54.597926 5798 catalog_manager.cc:1347] Generated new cluster ID: 4d3a089b1a1d4134a335f194cbe90d36
I20260620 12:19:54.598017 5798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:54.613409 5798 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:54.614382 5798 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:54.626752 5798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b0559f88f54a8bbdc64f07d69c0191: Generated new TSK 0
I20260620 12:19:54.627480 5798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:19:54.641721 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:54.636850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.0.98.62:41549)
0620 12:19:54.636987 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:54.636992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:54.637221 (+ 229us) server_negotiation.cc:407] Connection header received
0620 12:19:54.637379 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:54.637381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:54.637427 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:54.637514 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:19:54.637805 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.638314 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.639538 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.639751 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.639830 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.640141 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:19:54.641360 (+ 1219us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 12:19:54.641589 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39536: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 12:19:54.641789 5787 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39536: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 12:19:54.641804 5813 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:19:54.636792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.0.1:39536)
0620 12:19:54.637126 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:54.637141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:54.637253 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:54.637537 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:54.637540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:54.637549 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:19:54.637656 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:54.637663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.638431 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.638434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:54.639421 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:54.639431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.639846 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.639869 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:54.639963 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.639990 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:19:54.641515 (+ 1525us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 12:19:54.641645 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":124,"threads_started":1}
I20260620 12:19:54.642287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 5717
E20260620 12:19:54.654879 5814 webserver.cc:533] Webserver: sslize failed:
E20260620 12:19:54.654950 5814 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T12:19:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T12:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:54Z Disabled control of system clock
I20260620 12:19:54.690464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45555
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45555
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/testchainca.cert with env {}
W20260620 12:19:54.797590 5818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:54.797814 5818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:54.797887 5818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:54.801810 5818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:54.801857 5818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:54.801874 5818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:54.801890 5818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:54.801905 5818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 12:19:54.801920 5818 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 12:19:54.801951 5818 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 12:19:54.807140 5818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:54.808084 5818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:54.809034 5818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:54.817180 5823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:54.817199 5824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:54.817332 5826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:54.817606 5818 server_base.cc:1034] running on GCE node
I20260620 12:19:54.818217 5818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:54.819372 5818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:54.820585 5818 hybrid_clock.cc:648] HybridClock initialized: now 1781957994820568 us; error 33 us; skew 500 ppm
I20260620 12:19:54.822445 5818 webserver.cc:460] Webserver started at http://127.0.98.62:36657/ using document root <none> and password file <none>
I20260620 12:19:54.823047 5818 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:54.823140 5818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:54.823406 5818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:54.825469 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "9e2667ed99a44769b03be92d4edbafd5"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "c4b37b426fc70bbc51c85163d872de8b"
server_key_iv: "7199d4feacf37e66f40d164409d6fe5b"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.826016 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e2667ed99a44769b03be92d4edbafd5"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "c4b37b426fc70bbc51c85163d872de8b"
server_key_iv: "7199d4feacf37e66f40d164409d6fe5b"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.829545 5818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:54.831887 5832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:54.833189 5818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:54.833317 5818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "9e2667ed99a44769b03be92d4edbafd5"
format_stamp: "Formatted at 2026-06-20 12:19:54 on dist-test-slave-9b3m"
server_key: "c4b37b426fc70bbc51c85163d872de8b"
server_key_iv: "7199d4feacf37e66f40d164409d6fe5b"
server_key_version: "encryptionkey@0"
I20260620 12:19:54.833473 5818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:54.849733 5818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:54.850348 5818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:54.850534 5818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:54.857306 5818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45555
I20260620 12:19:54.857312 5884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45555 every 8 connection(s)
I20260620 12:19:54.858189 5818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:54.861136 5885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:54.865531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 5818
I20260620 12:19:54.865691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:54.865957 392 external_mini_cluster.cc:1428] Setting key ee99516845ed21967be27b49f258f4a1
I20260620 12:19:54.866596 5885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5: Bootstrap starting.
I20260620 12:19:54.868937 5885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:54.869688 5885 log.cc:824] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:54.871563 5885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5: No bootstrap required, opened a new log
I20260620 12:19:54.875336 5885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2667ed99a44769b03be92d4edbafd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45555 } }
I20260620 12:19:54.875577 5885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:54.875664 5885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2667ed99a44769b03be92d4edbafd5, State: Initialized, Role: FOLLOWER
I20260620 12:19:54.875800 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:54.867319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.0.98.62:45555)
0620 12:19:54.867987 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:54.868001 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:54.868038 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:19:54.868774 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:54.868793 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:54.869313 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:54.869674 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:54.870174 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.871327 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.872359 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.872853 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.873030 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.873995 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:54.874000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:54.874012 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:19:54.874398 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:54.874452 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:19:54.874471 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:54.874583 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:54.874729 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 12:19:54.874971 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":129,"threads_started":1}
I20260620 12:19:54.876225 5885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2667ed99a44769b03be92d4edbafd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45555 } }
I20260620 12:19:54.876382 5885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:54.876458 5885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:54.876546 5885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:54.877512 5885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2667ed99a44769b03be92d4edbafd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45555 } }
I20260620 12:19:54.877856 5885 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2667ed99a44769b03be92d4edbafd5; no voters:
I20260620 12:19:54.878162 5885 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:54.878245 5890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:54.878489 5890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2667ed99a44769b03be92d4edbafd5, State: Running, Role: LEADER
I20260620 12:19:54.878640 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:19:54.878788 5890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2667ed99a44769b03be92d4edbafd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45555 } }
I20260620 12:19:54.879360 5885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:54.880353 5891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e2667ed99a44769b03be92d4edbafd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2667ed99a44769b03be92d4edbafd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45555 } } }
I20260620 12:19:54.880481 5891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:54.880885 5899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:54.880867 5892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2667ed99a44769b03be92d4edbafd5. Latest consensus state: current_term: 1 leader_uuid: "9e2667ed99a44769b03be92d4edbafd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2667ed99a44769b03be92d4edbafd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45555 } } }
I20260620 12:19:54.881129 5892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:54.883972 5899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:54.889845 5899 catalog_manager.cc:1347] Generated new cluster ID: 74c364dd47714a209330c671c2441c9e
I20260620 12:19:54.889941 5899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:54.900214 5899 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:54.901274 5899 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:54.908655 5899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e2667ed99a44769b03be92d4edbafd5: Generated new TSK 0
I20260620 12:19:54.909427 5899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:19:55.089789 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:54.926047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.0.98.62:45555)
0620 12:19:54.926200 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:54.926207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:54.926382 (+ 175us) server_negotiation.cc:407] Connection header received
0620 12:19:54.926567 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:54.926570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:54.926616 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:54.926709 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0620 12:19:54.926958 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.927526 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.928734 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.928943 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.929020 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.929357 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 12:19:55.089207 (+159850us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 12:19:55.089631 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33102: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 12:19:55.089892 5888 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33102: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 12:19:55.089919 5914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:19:54.925960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45555 (local address 127.0.0.1:33102)
0620 12:19:54.926280 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:54.926295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:54.926414 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:54.926735 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:54.926740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:54.926749 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 12:19:54.926854 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:54.926861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.927633 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.927637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:54.928621 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:54.928631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:54.929062 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:54.929083 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:54.929171 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:54.929197 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 12:19:55.089527 (+160330us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 12:19:55.089719 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45555: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":128,"threads_started":1}
I20260620 12:19:55.090708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 5818
2026-06-20T12:19:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (427 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:55 dist-test-slave-9b3m krb5kdc[5926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:55 dist-test-slave-9b3m krb5kdc[5926](info): set up 2 sockets
Jun 20 12:19:55 dist-test-slave-9b3m krb5kdc[5926](info): commencing operation
krb5kdc: starting...
W20260620 12:19:56.489408 392 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:56 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:19:56.609027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39041
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:56.714381 5939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:56.714655 5939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:56.714730 5939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:56.718688 5939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:56.718777 5939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:56.718816 5939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:56.718874 5939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:56.718914 5939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:56.723927 5939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:56.724895 5939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:56.725873 5939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:56.734027 5945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:56.734054 5944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:56.734068 5947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:56.734366 5939 server_base.cc:1034] running on GCE node
I20260620 12:19:56.735035 5939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:56.736099 5939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:56.737282 5939 hybrid_clock.cc:648] HybridClock initialized: now 1781957996737267 us; error 29 us; skew 500 ppm
Jun 20 12:19:56 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:56.743332 5939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:19:56.744215 5939 webserver.cc:460] Webserver started at http://127.0.98.62:39681/ using document root <none> and password file <none>
I20260620 12:19:56.744776 5939 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:56.744863 5939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:56.745127 5939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:56.747126 5939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "0a333713a6664a86bb9b2a3c60402971"
format_stamp: "Formatted at 2026-06-20 12:19:56 on dist-test-slave-9b3m"
server_key: "a9813954514a5f3e8f261036b93cff45"
server_key_iv: "ac3b327af39309b74924a393bde77266"
server_key_version: "encryptionkey@0"
I20260620 12:19:56.747660 5939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a333713a6664a86bb9b2a3c60402971"
format_stamp: "Formatted at 2026-06-20 12:19:56 on dist-test-slave-9b3m"
server_key: "a9813954514a5f3e8f261036b93cff45"
server_key_iv: "ac3b327af39309b74924a393bde77266"
server_key_version: "encryptionkey@0"
I20260620 12:19:56.751271 5939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:56.753615 5954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:56.754971 5939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:56.755101 5939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "0a333713a6664a86bb9b2a3c60402971"
format_stamp: "Formatted at 2026-06-20 12:19:56 on dist-test-slave-9b3m"
server_key: "a9813954514a5f3e8f261036b93cff45"
server_key_iv: "ac3b327af39309b74924a393bde77266"
server_key_version: "encryptionkey@0"
I20260620 12:19:56.755204 5939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:56.771184 5939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:56.774250 5939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:56.774453 5939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:56.781217 5939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39041
I20260620 12:19:56.781498 6006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39041 every 8 connection(s)
I20260620 12:19:56.782160 5939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:56.784821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 5939
I20260620 12:19:56.784849 6007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:56.784948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:56.785260 392 external_mini_cluster.cc:1428] Setting key 83ab137e7b607514a50c3a1c9316d56f
I20260620 12:19:56.791419 6007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971: Bootstrap starting.
I20260620 12:19:56.794102 6007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971: Neither blocks nor log segments found. Creating new log.
Jun 20 12:19:56 dist-test-slave-9b3m krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:56.794840 6007 log.cc:824] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:56.797264 6007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971: No bootstrap required, opened a new log
I20260620 12:19:56.800364 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:56.786635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.0.98.62:39041)
0620 12:19:56.787101 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:56.787119 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:56.787156 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:19:56.787981 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:56.788000 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:56.788372 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:56.788660 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:56.789573 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:56.790514 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:56.791685 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:56.792155 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:56.792276 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:56.795097 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:56.795104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:56.795120 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:56.795153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:56.797167 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:56.797687 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:56.797693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:56.797698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:56.797829 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:56.798132 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:56.798135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:56.798136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:56.798600 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:56.798791 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:56.799064 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 12:19:56.799380 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":146,"threads_started":1}
I20260620 12:19:56.801647 6007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a333713a6664a86bb9b2a3c60402971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39041 } }
I20260620 12:19:56.801855 6007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:56.801915 6007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a333713a6664a86bb9b2a3c60402971, State: Initialized, Role: FOLLOWER
I20260620 12:19:56.802376 6007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a333713a6664a86bb9b2a3c60402971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39041 } }
I20260620 12:19:56.802533 6007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:56.802595 6007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:56.802702 6007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:56.803521 6007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a333713a6664a86bb9b2a3c60402971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39041 } }
I20260620 12:19:56.803849 6007 leader_election.cc:302] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a333713a6664a86bb9b2a3c60402971; no voters:
I20260620 12:19:56.804127 6007 leader_election.cc:288] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:56.804280 6012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:56.804520 6012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [term 1 LEADER]: Becoming Leader. State: Replica: 0a333713a6664a86bb9b2a3c60402971, State: Running, Role: LEADER
I20260620 12:19:56.804786 6012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a333713a6664a86bb9b2a3c60402971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39041 } }
I20260620 12:19:56.805445 6007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:56.806712 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a333713a6664a86bb9b2a3c60402971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a333713a6664a86bb9b2a3c60402971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39041 } } }
I20260620 12:19:56.806720 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a333713a6664a86bb9b2a3c60402971. Latest consensus state: current_term: 1 leader_uuid: "0a333713a6664a86bb9b2a3c60402971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a333713a6664a86bb9b2a3c60402971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39041 } } }
I20260620 12:19:56.806851 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:56.806851 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971 [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:56.807346 6022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:56.810333 6022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:56.816002 6022 catalog_manager.cc:1347] Generated new cluster ID: 621a8514e92549a2be9c0907312bd261
I20260620 12:19:56.816066 6022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:56.823437 6022 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:56.824572 6022 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:56.833776 6022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a333713a6664a86bb9b2a3c60402971: Generated new TSK 0
I20260620 12:19:56.834492 6022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:19:56.867761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39041
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:56.976483 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:56.976753 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:56.976827 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:56.980749 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:56.980832 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:56.980921 6033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:56.985989 6033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:56.986936 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:56.987906 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:56.997238 6039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:56.997287 6038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:56.997210 6041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:56.997373 6033 server_base.cc:1034] running on GCE node
I20260620 12:19:56.997788 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:56.998379 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:56.999544 6033 hybrid_clock.cc:648] HybridClock initialized: now 1781957996999532 us; error 26 us; skew 500 ppm
Jun 20 12:19:56 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.002236 6033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:19:57.003108 6033 webserver.cc:460] Webserver started at http://127.0.98.1:46843/ using document root <none> and password file <none>
I20260620 12:19:57.003594 6033 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:57.003650 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:57.003798 6033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:57.005790 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c9cfdf0dbd849468caf61ac12ceecc3"
format_stamp: "Formatted at 2026-06-20 12:19:56 on dist-test-slave-9b3m"
server_key: "c8f190ad7c8d22be8a9adf12c024a48a"
server_key_iv: "2554337750b5ffaa66d9cdddb52c8547"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.006346 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c9cfdf0dbd849468caf61ac12ceecc3"
format_stamp: "Formatted at 2026-06-20 12:19:56 on dist-test-slave-9b3m"
server_key: "c8f190ad7c8d22be8a9adf12c024a48a"
server_key_iv: "2554337750b5ffaa66d9cdddb52c8547"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.009814 6033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:57.012097 6048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.013375 6033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 12:19:57.013489 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "8c9cfdf0dbd849468caf61ac12ceecc3"
format_stamp: "Formatted at 2026-06-20 12:19:56 on dist-test-slave-9b3m"
server_key: "c8f190ad7c8d22be8a9adf12c024a48a"
server_key_iv: "2554337750b5ffaa66d9cdddb52c8547"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.013582 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:57.028749 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:57.031441 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:57.031633 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:57.032225 6033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:57.032976 6033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:57.033021 6033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.033143 6033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:57.033177 6033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.041904 6033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33253
I20260620 12:19:57.041931 6161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33253 every 8 connection(s)
I20260620 12:19:57.042728 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:57.043654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 6033
I20260620 12:19:57.043781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:57.044073 392 external_mini_cluster.cc:1428] Setting key e2dbba8756a70894a0b0f538ea0e8ea0
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:57.057271 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.044556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52105 (local address 127.0.98.62:39041)
0620 12:19:57.044703 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.044708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.045609 (+ 901us) server_negotiation.cc:407] Connection header received
0620 12:19:57.047589 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.047594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.047654 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.047750 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.048807 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.049414 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.050609 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.051027 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.051152 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.053576 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:57.053583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:57.053587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:57.053617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:57.055079 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.055602 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.055608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.055615 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.055687 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.056021 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.056026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.056028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.056202 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:57.056373 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:57.056881 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 12:19:57.057010 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:19:57.058231 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.044945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39041 (local address 127.0.98.1:52105)
0620 12:19:57.045443 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:57.045507 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:57.047346 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:57.047870 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:57.047876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:57.048201 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:57.048666 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:57.048678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.049533 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.049537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:57.050483 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:57.050492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.051179 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.051182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:57.051351 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.051983 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:57.051997 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:57.053415 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:57.055179 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:57.055184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:57.055194 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:57.055487 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:57.055790 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:57.055793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:57.055795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:57.055925 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:57.056345 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:57.056352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:57.056645 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 12:19:57.056840 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 12:19:57.057168 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":187,"threads_started":1}
I20260620 12:19:57.059456 6162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39041
I20260620 12:19:57.059710 6162 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:57.060230 6162 heartbeater.cc:511] Master 127.0.98.62:39041 requested a full tablet report, sending...
I20260620 12:19:57.061654 5971 ts_manager.cc:194] Registered new tserver with Master: 8c9cfdf0dbd849468caf61ac12ceecc3 (127.0.98.1:33253)
I20260620 12:19:57.062812 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:19:57.075598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39041
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:57.182189 6167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:57.182446 6167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:57.182516 6167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:57.186478 6167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:57.186549 6167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:57.186633 6167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:57.191681 6167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:57.192628 6167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:57.193789 6167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:57.202922 6175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:57.203006 6173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:57.203208 6167 server_base.cc:1034] running on GCE node
W20260620 12:19:57.202934 6172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:57.203613 6167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:57.204198 6167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:57.205384 6167 hybrid_clock.cc:648] HybridClock initialized: now 1781957997205357 us; error 39 us; skew 500 ppm
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.208266 6167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:19:57.209148 6167 webserver.cc:460] Webserver started at http://127.0.98.2:42621/ using document root <none> and password file <none>
I20260620 12:19:57.209705 6167 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:57.209810 6167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:57.210026 6167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:57.211922 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ffe6bd583bb4fc7858f778abceedb03"
format_stamp: "Formatted at 2026-06-20 12:19:57 on dist-test-slave-9b3m"
server_key: "ca58725045902779ed953505027689d1"
server_key_iv: "cec1af6c2da8bbb21d5eb8fddbd8845e"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.212459 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ffe6bd583bb4fc7858f778abceedb03"
format_stamp: "Formatted at 2026-06-20 12:19:57 on dist-test-slave-9b3m"
server_key: "ca58725045902779ed953505027689d1"
server_key_iv: "cec1af6c2da8bbb21d5eb8fddbd8845e"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.215940 6167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:19:57.218355 6182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.219535 6167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:19:57.219668 6167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "9ffe6bd583bb4fc7858f778abceedb03"
format_stamp: "Formatted at 2026-06-20 12:19:57 on dist-test-slave-9b3m"
server_key: "ca58725045902779ed953505027689d1"
server_key_iv: "cec1af6c2da8bbb21d5eb8fddbd8845e"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.219789 6167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:57.238272 6167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:57.241326 6167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:57.241545 6167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:57.242128 6167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:57.242924 6167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:57.242969 6167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.243007 6167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:57.243060 6167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.252146 6167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41163
I20260620 12:19:57.252169 6295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41163 every 8 connection(s)
I20260620 12:19:57.253026 6167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:57.261996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 6167
I20260620 12:19:57.262217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:57.262538 392 external_mini_cluster.cc:1428] Setting key e072587a6fba0d53c7bf1f2f285ca3fb
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:57.267081 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.254795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44327 (local address 127.0.98.62:39041)
0620 12:19:57.254920 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.254924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.255660 (+ 736us) server_negotiation.cc:407] Connection header received
0620 12:19:57.256998 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.257002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.257080 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.257156 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.258314 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.258984 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.260178 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.260416 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.260524 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.263418 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:57.263422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:57.263424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:57.263449 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:57.264922 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.265566 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.265572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.265576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.265669 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.266034 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.266037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.266039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.266213 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:57.266325 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:57.266814 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 12:19:57.266927 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:19:57.267889 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.255092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39041 (local address 127.0.98.2:44327)
0620 12:19:57.255516 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:57.255556 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:57.256793 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:57.257327 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:57.257334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:57.257674 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:57.258172 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:57.258186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.259103 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.259110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:57.260052 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:57.260062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.260564 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.260568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:57.260740 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.261456 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:57.261473 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:57.263266 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:57.265041 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:57.265066 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:57.265084 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:57.265466 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:57.265783 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:57.265786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:57.265789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:57.265927 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:57.266324 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:57.266331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:57.266569 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 12:19:57.266774 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 12:19:57.267090 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260620 12:19:57.269071 6296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39041
I20260620 12:19:57.269325 6296 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:57.269783 6296 heartbeater.cc:511] Master 127.0.98.62:39041 requested a full tablet report, sending...
I20260620 12:19:57.270802 5971 ts_manager.cc:194] Registered new tserver with Master: 9ffe6bd583bb4fc7858f778abceedb03 (127.0.98.2:41163)
I20260620 12:19:57.271607 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44327
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:19:57.290024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39041
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:57.398391 6301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:57.398629 6301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:57.398671 6301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:57.402904 6301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:57.402961 6301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:57.403048 6301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:57.408485 6301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:57.409516 6301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:57.410483 6301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:57.419787 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:57.419823 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:57.419934 6301 server_base.cc:1034] running on GCE node
W20260620 12:19:57.419795 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:57.420430 6301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:57.421134 6301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:57.422344 6301 hybrid_clock.cc:648] HybridClock initialized: now 1781957997422334 us; error 30 us; skew 500 ppm
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.425343 6301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:19:57.426172 6301 webserver.cc:460] Webserver started at http://127.0.98.3:45397/ using document root <none> and password file <none>
I20260620 12:19:57.426716 6301 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:57.426805 6301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:57.427026 6301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:57.428983 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfc5f201c4e049bfa4cdacf262db70f4"
format_stamp: "Formatted at 2026-06-20 12:19:57 on dist-test-slave-9b3m"
server_key: "7a5e98e1d2704892feeb0445d7aa4ec6"
server_key_iv: "841ab24be5a55e5f34beb99e53da8f88"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.429617 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfc5f201c4e049bfa4cdacf262db70f4"
format_stamp: "Formatted at 2026-06-20 12:19:57 on dist-test-slave-9b3m"
server_key: "7a5e98e1d2704892feeb0445d7aa4ec6"
server_key_iv: "841ab24be5a55e5f34beb99e53da8f88"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.433197 6301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:19:57.435554 6316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.436746 6301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:57.436890 6301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "dfc5f201c4e049bfa4cdacf262db70f4"
format_stamp: "Formatted at 2026-06-20 12:19:57 on dist-test-slave-9b3m"
server_key: "7a5e98e1d2704892feeb0445d7aa4ec6"
server_key_iv: "841ab24be5a55e5f34beb99e53da8f88"
server_key_version: "encryptionkey@0"
I20260620 12:19:57.437000 6301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:57.451097 6301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:57.454377 6301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:57.454582 6301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:57.455180 6301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:57.455987 6301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:57.456064 6301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.456132 6301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:57.456182 6301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:57.465404 6301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38553
I20260620 12:19:57.465454 6429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38553 every 8 connection(s)
I20260620 12:19:57.466296 6301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:57.475535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 6301
I20260620 12:19:57.475669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:19:57.475982 392 external_mini_cluster.cc:1428] Setting key 5074b2cbf85a62b8d4c12e6ffd8064ec
I20260620 12:19:57.480417 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.468179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58245 (local address 127.0.98.62:39041)
0620 12:19:57.468334 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.468337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.469162 (+ 825us) server_negotiation.cc:407] Connection header received
0620 12:19:57.470540 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.470543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.470599 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.470708 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.471848 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.472378 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.473549 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.473771 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.473885 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.476711 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:57.476714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:57.476717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:57.476744 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:57.478440 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.478948 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.478951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.478952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.479035 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.479391 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.479394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.479395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.479570 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:57.479655 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:57.480207 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 12:19:57.480302 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:19:57.481279 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.468543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39041 (local address 127.0.98.3:58245)
0620 12:19:57.468984 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:57.469028 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:57.470328 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:57.470886 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:57.470893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:57.471230 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:57.471683 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:57.471695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.472512 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.472521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:57.473412 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:57.473420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.473918 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.473922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:57.474086 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.474615 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:57.474632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:57.476517 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:57.478552 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:57.478556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:57.478568 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:57.478835 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:57.479139 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:57.479141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:57.479142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:57.479283 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:57.479689 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:57.479695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:57.479921 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 12:19:57.480104 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 12:19:57.480351 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260620 12:19:57.482337 6430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39041
I20260620 12:19:57.482544 6430 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:57.482962 6430 heartbeater.cc:511] Master 127.0.98.62:39041 requested a full tablet report, sending...
I20260620 12:19:57.483912 5971 ts_manager.cc:194] Registered new tserver with Master: dfc5f201c4e049bfa4cdacf262db70f4 (127.0.98.3:38553)
I20260620 12:19:57.484462 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58245
I20260620 12:19:57.489812 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/test-admin.keytab.
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.506808 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 12:19:57.509850 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:19:57.508303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.0.98.1:33253)
0620 12:19:57.508627 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.508637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.508660 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:19:57.508712 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.508717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.508896 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.509014 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.509658 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37168: BlockingRecv error: recv got EOF from 127.0.0.1:37168 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":133,"threads_started":1}
W20260620 12:19:57.509887 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.511366 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.512478 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:57.522398 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.513405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.0.98.1:33253)
0620 12:19:57.513569 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.513573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.513626 (+ 53us) server_negotiation.cc:407] Connection header received
0620 12:19:57.513740 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.513743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.513783 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.513856 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.514356 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.515218 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.516028 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.516439 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.516533 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.519010 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:57.519018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:57.519030 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:57.519053 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:57.520654 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.521108 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.521115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.521119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.521200 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.521473 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.521477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.521478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.521711 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:57.521912 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:57.522091 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 12:19:57.522249 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 12:19:57.526017 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:19:57.525140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.0.98.1:33253)
0620 12:19:57.525263 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.525266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.525383 (+ 117us) server_negotiation.cc:407] Connection header received
0620 12:19:57.525467 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.525469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.525501 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.525564 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.525905 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37174: BlockingRecv error: recv got EOF from 127.0.0.1:37174 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 12:19:57.526347 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5cc)
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.527290 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.528119 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:57.536821 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.528916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.0.98.1:33253)
0620 12:19:57.529058 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.529063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.529168 (+ 105us) server_negotiation.cc:407] Connection header received
0620 12:19:57.529306 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.529309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.529354 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.529434 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.529907 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.530419 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.531206 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.531514 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.531631 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.533954 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:57.533957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:57.533960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:57.533984 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:57.535301 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.535690 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.535695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.535697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.535776 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.536036 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.536040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.536042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.536204 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:57.536331 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:57.536558 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 12:19:57.536669 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 12:19:57.539398 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:19:57.538575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.0.98.1:33253)
0620 12:19:57.538663 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.538666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.538749 (+ 83us) server_negotiation.cc:407] Connection header received
0620 12:19:57.538834 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.538836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.538865 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.538926 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.539319 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37198: BlockingRecv error: recv got EOF from 127.0.0.1:37198 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 12:19:57.539497 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781957967886955-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.540498 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:57.541405 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:19:57.549408 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:57.542227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.0.98.1:33253)
0620 12:19:57.542328 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:57.542331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:57.542356 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:19:57.542450 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:57.542452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:57.542481 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:57.542543 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:57.542956 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.543420 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.544309 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:57.544501 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:57.544571 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:57.546596 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:57.546599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:57.546600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:57.546619 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:57.547942 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.548379 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.548381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.548382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.548448 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:57.548668 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:57.548670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:57.548671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:57.548805 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:57.548891 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:57.549112 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 12:19:57.549209 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:19:57.550287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 6033
I20260620 12:19:57.556006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 6167
I20260620 12:19:57.561762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 6301
I20260620 12:19:57.567051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 5939
2026-06-20T12:19:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2223 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[6450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[6450](info): set up 2 sockets
Jun 20 12:19:57 dist-test-slave-9b3m krb5kdc[6450](info): commencing operation
krb5kdc: starting...
W20260620 12:19:58.601362 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:19:58 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:19:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:19:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 12:19:58.719975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:58.827214 6463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:58.827522 6463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:58.827613 6463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:58.831660 6463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:58.831751 6463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:58.831794 6463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:58.831857 6463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:58.831907 6463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:58.837143 6463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:58.838179 6463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:58.839229 6463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:58.847669 6468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:58.847676 6469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:58.847811 6463 server_base.cc:1034] running on GCE node
W20260620 12:19:58.847676 6471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:58.848526 6463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:58.849706 6463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:58.850921 6463 hybrid_clock.cc:648] HybridClock initialized: now 1781957998850895 us; error 39 us; skew 500 ppm
Jun 20 12:19:58 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957998, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:58.853848 6463 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 12:19:58.854791 6463 webserver.cc:460] Webserver started at http://127.0.98.62:38741/ using document root <none> and password file <none>
I20260620 12:19:58.855319 6463 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:58.855381 6463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:58.855557 6463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:58.857698 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "f5e86be99fe3493c8908628297b224cc"
format_stamp: "Formatted at 2026-06-20 12:19:58 on dist-test-slave-9b3m"
server_key: "5d2cecc45ac45d023f5ec44ce1109fb3"
server_key_iv: "be40fdebc390c993b373b0d4dcae98e4"
server_key_version: "encryptionkey@0"
I20260620 12:19:58.858237 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5e86be99fe3493c8908628297b224cc"
format_stamp: "Formatted at 2026-06-20 12:19:58 on dist-test-slave-9b3m"
server_key: "5d2cecc45ac45d023f5ec44ce1109fb3"
server_key_iv: "be40fdebc390c993b373b0d4dcae98e4"
server_key_version: "encryptionkey@0"
I20260620 12:19:58.862053 6463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:19:58.864540 6478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:58.865900 6463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:58.866035 6463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "f5e86be99fe3493c8908628297b224cc"
format_stamp: "Formatted at 2026-06-20 12:19:58 on dist-test-slave-9b3m"
server_key: "5d2cecc45ac45d023f5ec44ce1109fb3"
server_key_iv: "be40fdebc390c993b373b0d4dcae98e4"
server_key_version: "encryptionkey@0"
I20260620 12:19:58.866154 6463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:58.875964 6463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:58.878835 6463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:58.879047 6463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:58.886328 6463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37687
I20260620 12:19:58.886606 6530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37687 every 8 connection(s)
I20260620 12:19:58.887372 6463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:19:58.890154 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:58.895366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 6463
I20260620 12:19:58.895499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:19:58.895525 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: Bootstrap starting.
I20260620 12:19:58.895778 392 external_mini_cluster.cc:1428] Setting key 7706c6ee70ee77281574ee66cb3ab599
I20260620 12:19:58.898061 6531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:58.898761 6531 log.cc:824] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:58.900635 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: No bootstrap required, opened a new log
Jun 20 12:19:58 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:19:58.904062 6531 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } }
I20260620 12:19:58.904314 6531 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:58.904392 6531 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e86be99fe3493c8908628297b224cc, State: Initialized, Role: FOLLOWER
I20260620 12:19:58.904850 6531 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } }
I20260620 12:19:58.905009 6531 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:19:58.905107 6531 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:19:58.905213 6531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:58.906230 6531 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } }
I20260620 12:19:58.906571 6531 leader_election.cc:302] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5e86be99fe3493c8908628297b224cc; no voters:
I20260620 12:19:58.906890 6531 leader_election.cc:288] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:19:58.906947 6536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:19:58.907120 6536 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 1 LEADER]: Becoming Leader. State: Replica: f5e86be99fe3493c8908628297b224cc, State: Running, Role: LEADER
I20260620 12:19:58.907452 6536 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } }
I20260620 12:19:58.908088 6531 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:19:58.908809 6538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } }
I20260620 12:19:58.908942 6538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:58.909216 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e86be99fe3493c8908628297b224cc. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } }
I20260620 12:19:58.909381 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:58.911661 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:58.897150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.0.98.62:37687)
0620 12:19:58.897557 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:58.897571 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:58.897606 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:19:58.898269 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:58.898289 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:58.898630 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:58.898935 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:58.899705 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:58.900665 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:58.901575 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:58.902038 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:58.902205 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:58.904791 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:58.904795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:58.904807 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:58.904832 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:58.906917 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:58.907451 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:58.907457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:58.907462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:58.907611 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:58.907899 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:58.907904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:58.907907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:58.908410 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:58.908649 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:58.910631 (+ 1982us) server_negotiation.cc:299] Negotiation successful
0620 12:19:58.910903 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":130,"threads_started":1}
W20260620 12:19:58.912983 6551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:19:58.913091 6551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:19:58.913203 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:19:58.914079 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:19:58.918936 6545 catalog_manager.cc:1347] Generated new cluster ID: 24156dbf37c0475cb36282614b8402d0
I20260620 12:19:58.919006 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:19:58.932354 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:19:58.933323 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:19:58.944002 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: Generated new TSK 0
I20260620 12:19:58.944644 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 12:19:58.972726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37687
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:59.080775 6557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:59.081014 6557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:59.081146 6557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:59.085160 6557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:59.085210 6557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:59.085266 6557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:19:59.090479 6557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:59.091461 6557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:59.092454 6557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:59.101464 6562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:59.101575 6557 server_base.cc:1034] running on GCE node
W20260620 12:19:59.101487 6563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:59.101501 6565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:59.102142 6557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:59.102757 6557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:59.103960 6557 hybrid_clock.cc:648] HybridClock initialized: now 1781957999103934 us; error 43 us; skew 500 ppm
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:59.106739 6557 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 12:19:59.107568 6557 webserver.cc:460] Webserver started at http://127.0.98.1:45193/ using document root <none> and password file <none>
I20260620 12:19:59.108151 6557 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:59.108246 6557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:59.108505 6557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:59.110800 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "7ad5a9dc70444049a8340f57f8928cd7"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "fec58798cf12eb50d066dc29d6b140bb"
server_key_iv: "640dae1fa4d8c1a304ab25568753b9f4"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.111339 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7ad5a9dc70444049a8340f57f8928cd7"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "fec58798cf12eb50d066dc29d6b140bb"
server_key_iv: "640dae1fa4d8c1a304ab25568753b9f4"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.115000 6557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:19:59.117318 6572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.118642 6557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:59.118736 6557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "7ad5a9dc70444049a8340f57f8928cd7"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "fec58798cf12eb50d066dc29d6b140bb"
server_key_iv: "640dae1fa4d8c1a304ab25568753b9f4"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.118813 6557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:59.141232 6557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:59.144414 6557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:59.144570 6557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:59.145196 6557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:59.146016 6557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:59.146061 6557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.146101 6557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:59.146116 6557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.154541 6557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35671
I20260620 12:19:59.154573 6685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35671 every 8 connection(s)
I20260620 12:19:59.155380 6557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:19:59.158689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 6557
I20260620 12:19:59.158835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:19:59.159174 392 external_mini_cluster.cc:1428] Setting key d4efadb2e538c17afa4cf603fc9b6a91
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:19:59.171125 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.157120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55137 (local address 127.0.98.62:37687)
0620 12:19:59.157235 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.157239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.158099 (+ 860us) server_negotiation.cc:407] Connection header received
0620 12:19:59.159804 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.159807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.159862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.159966 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.161352 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.162189 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.163979 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.164220 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.164303 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.167174 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.167177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.167179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.167201 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.168745 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.169374 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.169377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.169379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.169482 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.169832 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.169835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.169837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.170121 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.170233 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.170800 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.170964 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 12:19:59.172025 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.157526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:55137)
0620 12:19:59.157944 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.157984 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.159597 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.160136 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.160143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.160474 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.161174 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.161198 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.162348 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.162354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.163848 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.163863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.164372 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.164379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.164617 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.165319 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.165342 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.166997 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.168894 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.168899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.168913 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.169282 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.169583 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.169586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.169588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.169728 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.170231 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.170237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.170534 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.170714 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.171027 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260620 12:19:59.173233 6686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37687
I20260620 12:19:59.173506 6686 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:59.174081 6686 heartbeater.cc:511] Master 127.0.98.62:37687 requested a full tablet report, sending...
I20260620 12:19:59.175643 6495 ts_manager.cc:194] Registered new tserver with Master: 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671)
I20260620 12:19:59.177261 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 12:19:59.187885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37687
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:59.293748 6691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:59.293977 6691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:59.294014 6691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:59.297808 6691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:59.297850 6691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:59.297904 6691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:19:59.302956 6691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:59.303884 6691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:59.304823 6691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:59.314759 6697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:59.314750 6696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:59.314781 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:59.315019 6691 server_base.cc:1034] running on GCE node
I20260620 12:19:59.315465 6691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:59.316113 6691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:59.317298 6691 hybrid_clock.cc:648] HybridClock initialized: now 1781957999317277 us; error 31 us; skew 500 ppm
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:59.319960 6691 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 12:19:59.320716 6691 webserver.cc:460] Webserver started at http://127.0.98.2:39617/ using document root <none> and password file <none>
I20260620 12:19:59.321280 6691 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:59.321341 6691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:59.321545 6691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:59.323544 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "29f788b0584444fcaa8c693eabd2817a"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "9cf41b188c40403e2b33e64967ef35fc"
server_key_iv: "9c0ca17e40379862fdde32b52b8e5ea8"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.324038 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "29f788b0584444fcaa8c693eabd2817a"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "9cf41b188c40403e2b33e64967ef35fc"
server_key_iv: "9c0ca17e40379862fdde32b52b8e5ea8"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.327615 6691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:19:59.329988 6706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.331243 6691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 12:19:59.331390 6691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "29f788b0584444fcaa8c693eabd2817a"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "9cf41b188c40403e2b33e64967ef35fc"
server_key_iv: "9c0ca17e40379862fdde32b52b8e5ea8"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.331498 6691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:59.363428 6691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:59.366564 6691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:59.366782 6691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:59.367408 6691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:59.368256 6691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:59.368306 6691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.368376 6691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:59.368412 6691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.378217 6691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43129
I20260620 12:19:59.378247 6819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43129 every 8 connection(s)
I20260620 12:19:59.379150 6691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:19:59.383754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 6691
I20260620 12:19:59.383908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:19:59.384239 392 external_mini_cluster.cc:1428] Setting key b6de3132a66a6a140119cc634dc51fd6
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:19:59.394709 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.380924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33089 (local address 127.0.98.62:37687)
0620 12:19:59.381119 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.381124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.382092 (+ 968us) server_negotiation.cc:407] Connection header received
0620 12:19:59.383434 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.383437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.383499 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.383582 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.384935 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.385797 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.387606 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.388104 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.388239 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.390646 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.390653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.390661 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.390698 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.392528 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.393007 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.393011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.393013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.393156 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.393490 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.393496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.393498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.393757 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.393885 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.394349 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.394509 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:19:59.395345 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.381461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:33089)
0620 12:19:59.381923 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.381965 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.383223 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.383746 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.383753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.384200 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.384728 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.384742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.385950 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.385953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.387452 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.387466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.388259 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.388262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.388404 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.389020 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.389065 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.390471 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.392646 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.392650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.392661 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.392902 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.393256 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.393258 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.393260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.393384 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.393869 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.393874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.394092 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.394250 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.394485 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260620 12:19:59.396652 6820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37687
I20260620 12:19:59.396884 6820 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:59.397393 6820 heartbeater.cc:511] Master 127.0.98.62:37687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260620 12:19:59.398452 6495 ts_manager.cc:194] Registered new tserver with Master: 29f788b0584444fcaa8c693eabd2817a (127.0.98.2:43129)
I20260620 12:19:59.399029 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 12:19:59.413187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37687
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:59.520443 6825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:59.520722 6825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:59.520795 6825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:59.525458 6825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:59.525537 6825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:59.525624 6825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:19:59.530735 6825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:19:59.531790 6825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:59.532794 6825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:59.542301 6830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:59.542435 6825 server_base.cc:1034] running on GCE node
W20260620 12:19:59.542279 6833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:59.542300 6831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:59.542953 6825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:59.543499 6825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:59.544694 6825 hybrid_clock.cc:648] HybridClock initialized: now 1781957999544685 us; error 29 us; skew 500 ppm
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:59.547349 6825 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 12:19:59.548211 6825 webserver.cc:460] Webserver started at http://127.0.98.3:38561/ using document root <none> and password file <none>
I20260620 12:19:59.548758 6825 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:59.548844 6825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:59.549119 6825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:59.551056 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "c6253039999647e68e450a5d4993cfaf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "bebf8d250c0e66b18b292e9cf14dab75"
server_key_iv: "c2fec1b3adb7422de2f2e324730d8741"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.551625 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c6253039999647e68e450a5d4993cfaf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "bebf8d250c0e66b18b292e9cf14dab75"
server_key_iv: "c2fec1b3adb7422de2f2e324730d8741"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.555142 6825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 12:19:59.557601 6840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.558920 6825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:19:59.559065 6825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "c6253039999647e68e450a5d4993cfaf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "bebf8d250c0e66b18b292e9cf14dab75"
server_key_iv: "c2fec1b3adb7422de2f2e324730d8741"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.559183 6825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:59.591868 6825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:59.594880 6825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:59.595108 6825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:59.595752 6825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:19:59.596606 6825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:19:59.596683 6825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.596760 6825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:19:59.596818 6825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.606344 6825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33747
I20260620 12:19:59.606364 6953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33747 every 8 connection(s)
I20260620 12:19:59.607257 6825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:19:59.608980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 6825
I20260620 12:19:59.609127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:19:59.609463 392 external_mini_cluster.cc:1428] Setting key 9495a70f26244c9ba10304b6db67815f
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:19:59.622043 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.609829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57107 (local address 127.0.98.62:37687)
0620 12:19:59.609986 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.609993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.610569 (+ 576us) server_negotiation.cc:407] Connection header received
0620 12:19:59.612159 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.612163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.612217 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.612296 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.613861 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.614673 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.616323 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.616551 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.616644 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.619017 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.619020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.619022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.619045 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.620254 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.620712 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.620715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.620716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.620788 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.621127 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.621129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.621130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.621296 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.621374 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.621808 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.621903 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:19:59.623103 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.609899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:57107)
0620 12:19:59.610436 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.610476 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.611939 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.612521 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.612528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.613067 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.613722 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.613738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.614824 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.614828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.616200 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.616210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.616685 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.616688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.616823 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.617456 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.617471 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.618845 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.620360 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.620364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.620390 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.620625 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.620867 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.620869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.620871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.620998 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.621393 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.621397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.621603 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.621744 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.622024 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":142,"threads_started":1}
I20260620 12:19:59.624202 6954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37687
I20260620 12:19:59.624454 6954 heartbeater.cc:464] Registering TS with master...
I20260620 12:19:59.624940 6954 heartbeater.cc:511] Master 127.0.98.62:37687 requested a full tablet report, sending...
I20260620 12:19:59.625993 6495 ts_manager.cc:194] Registered new tserver with Master: c6253039999647e68e450a5d4993cfaf (127.0.98.3:33747)
I20260620 12:19:59.626519 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57107
I20260620 12:19:59.635229 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 12:19:59.660398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43043
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37687,127.0.98.61:43043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:19:59.768009 6959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:19:59.768271 6959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:19:59.768342 6959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:19:59.772234 6959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:19:59.772317 6959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:19:59.772356 6959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:19:59.772413 6959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:19:59.772450 6959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:19:59.777566 6959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37687,127.0.98.61:43043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
W20260620 12:19:59.777791 6959 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37687,127.0.98.61:43043'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 12:19:59.778582 6959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:19:59.779538 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:59.787840 6967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:59.787889 6965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:59.787947 6959 server_base.cc:1034] running on GCE node
W20260620 12:19:59.787896 6964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:59.788631 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:59.789708 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:59.790860 6959 hybrid_clock.cc:648] HybridClock initialized: now 1781957999790833 us; error 38 us; skew 500 ppm
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:59.793519 6959 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 12:19:59.794337 6959 webserver.cc:460] Webserver started at http://127.0.98.61:44495/ using document root <none> and password file <none>
I20260620 12:19:59.794866 6959 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:59.794955 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:59.795182 6959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:19:59.797150 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data/instance:
uuid: "e06b3f42475f46e4b74cc800eac7b1cf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "718978327de1d7488e3eb9e1e7a0b392"
server_key_iv: "d5677c8c4e632e81c739a68c2b658db0"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.797688 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal/instance:
uuid: "e06b3f42475f46e4b74cc800eac7b1cf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "718978327de1d7488e3eb9e1e7a0b392"
server_key_iv: "d5677c8c4e632e81c739a68c2b658db0"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.801158 6959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 12:19:59.803301 6974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.804425 6959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:19:59.804574 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
uuid: "e06b3f42475f46e4b74cc800eac7b1cf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "718978327de1d7488e3eb9e1e7a0b392"
server_key_iv: "d5677c8c4e632e81c739a68c2b658db0"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.804677 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:59.829195 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:19:59.831985 6959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:19:59.832187 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:19:59.839286 6959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43043
I20260620 12:19:59.839560 7026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43043 every 8 connection(s)
I20260620 12:19:59.840240 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:19:59.842911 7027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:19:59.845875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 6959
I20260620 12:19:59.846560 7027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:19:59.858319 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.848395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.0.98.62:37687)
0620 12:19:59.848545 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.848550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.848821 (+ 271us) server_negotiation.cc:407] Connection header received
0620 12:19:59.849030 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.849035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.849106 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.849197 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.850060 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.850560 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.851604 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.851814 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.851889 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.854574 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.854580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.854586 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.854623 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.856307 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.856811 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.856813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.856814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.856883 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.857263 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.857268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.857270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.857479 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.857601 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.858042 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.858152 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:19:59.859099 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.847284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.0.98.61:43043)
0620 12:19:59.847656 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.847672 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.847711 (+ 39us) server_negotiation.cc:407] Connection header received
0620 12:19:59.848427 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.848449 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.848855 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.849120 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.849823 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.850792 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.851610 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.852023 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.852116 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.855129 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.855136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.855147 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.855175 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.857035 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.857442 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.857447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.857449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.857538 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.857809 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.857814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.857816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.858040 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.858233 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.858816 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.858917 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":245,"spinlock_wait_cycles":13184,"thread_start_us":104,"threads_started":1}
I20260620 12:19:59.859102 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.848335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44022)
0620 12:19:59.848679 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.848725 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.848847 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.849251 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.849256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.849626 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.849919 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.849930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.850676 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.850680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.851465 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.851473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.851939 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.851943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.852110 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.852866 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.852881 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.854413 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.856407 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.856413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.856423 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.856707 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.856967 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.856971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.856973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.857170 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.857596 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.857603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.857821 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.858009 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.858280 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":54,"thread_start_us":115,"threads_started":1}
I20260620 12:19:59.860415 7027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:19:59.867389 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.860806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.0.1:57146)
0620 12:19:59.860909 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.860922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.861000 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.861229 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.861233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.861369 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.861447 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.861452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.862119 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.862121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.862677 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.862687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.863084 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.863087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.863157 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.863507 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.863517 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.864702 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.866278 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.866280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.866281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.866453 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.866691 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.866693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.866694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.866760 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.867042 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.867044 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.867108 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.867182 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.867266 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 12:19:59.867414 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.860892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.0.98.61:43043)
0620 12:19:59.860998 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.861002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.861017 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:19:59.861086 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.861089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.861126 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.861197 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.861552 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.862003 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.862778 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.862994 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.863072 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.864813 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.864816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.864818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.864836 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.866187 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.866545 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.866547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.866548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.866617 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.866830 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.866831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.866833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.866967 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.867038 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.867230 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.867307 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 12:19:59.870360 7027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Bootstrap starting.
I20260620 12:19:59.872452 7027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Neither blocks nor log segments found. Creating new log.
I20260620 12:19:59.873193 7027 log.cc:824] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Log is configured to *not* fsync() on all Append() calls
I20260620 12:19:59.874857 7027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: No bootstrap required, opened a new log
I20260620 12:19:59.877688 7027 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } }
I20260620 12:19:59.877892 7027 raft_consensus.cc:374] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:19:59.877974 7027 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06b3f42475f46e4b74cc800eac7b1cf, State: Initialized, Role: FOLLOWER
I20260620 12:19:59.878376 7027 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } }
I20260620 12:19:59.878808 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } } }
I20260620 12:19:59.878912 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:19:59.879451 7027 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: configured and running, proceeding with master startup.
W20260620 12:19:59.883894 7044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:19:59.883983 7044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:19:59.886301 6959 master_runner.cc:428] Detected that this master e06b3f42475f46e4b74cc800eac7b1cf is joining an existing cluster
I20260620 12:19:59.886363 6959 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43043
I20260620 12:19:59.902336 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.893944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.0.1:57154)
0620 12:19:59.894206 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.894218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.894328 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.894574 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.894576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.894796 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.894898 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.894903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.895807 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.895810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.896641 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.896649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.897293 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.897296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.897375 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.897901 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.897914 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.898317 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.900824 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.900830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.900834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.901152 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.901411 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.901414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.901415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.901492 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.901851 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.901854 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.901932 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.902029 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.902148 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260620 12:19:59.902351 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.893792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.0.98.61:43043)
0620 12:19:59.893892 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.893896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.894282 (+ 386us) server_negotiation.cc:407] Connection header received
0620 12:19:59.894408 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.894411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.894440 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.894479 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.894993 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.895694 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.896747 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.897096 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.897160 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.898453 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.898457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.898460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.898487 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.900731 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.901239 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.901243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.901245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.901324 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.901592 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.901595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.901597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.901775 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.901868 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.902094 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.902212 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 12:19:59.903162 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.893353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44024)
0620 12:19:59.893746 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.893760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.893890 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.894260 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.894264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.894513 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.894645 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.894654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.895965 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.895968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.896884 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.896893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.897613 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.897617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.897696 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.898467 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.898502 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.898933 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.900776 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.900782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.900787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.901150 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.901628 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.901631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.901633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.901712 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.902228 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.902233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.902308 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.902877 (+ 569us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.903002 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"spinlock_wait_cycles":30080,"thread_start_us":101,"threads_started":1}
I20260620 12:19:59.903167 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.893457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.0.98.62:37687)
0620 12:19:59.893580 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.893584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.893824 (+ 240us) server_negotiation.cc:407] Connection header received
0620 12:19:59.894007 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.894010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.894060 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.894111 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.894804 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.895660 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.897067 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.897407 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.897489 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.899068 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.899072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.899074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.899098 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.900661 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.901439 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.901442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.901453 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.901531 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.901810 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.901816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.901819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.902094 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.902189 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.902933 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.903036 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957998, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:19:59.917374 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.906997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.0.98.61:43043)
0620 12:19:59.907109 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.907112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.907878 (+ 766us) server_negotiation.cc:407] Connection header received
0620 12:19:59.908081 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.908083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.908144 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.908228 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.908985 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.909518 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.910724 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.910942 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.911024 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.913894 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.913897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.913900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.913923 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.915518 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.916041 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.916044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.916046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.916122 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.916405 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.916408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.916409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.916600 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.916665 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.917125 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.917229 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:19:59.918081 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.906960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.0.1:57170)
0620 12:19:59.907759 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.907806 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.907932 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.908278 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.908284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.908760 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.908876 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.908884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.909600 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.909604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.910590 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.910600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.911219 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.911223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.911330 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.911898 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.911912 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.913749 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.915640 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.915648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.915654 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.915928 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.916225 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.916228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.916230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.916310 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.916759 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.916765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.916877 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.917731 (+ 854us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.917919 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":73,"threads_started":1}
I20260620 12:19:59.919781 6495 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43043: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43043
}
attrs {
promote: true
}
}
dest_uuid: "f5e86be99fe3493c8908628297b224cc"
cas_config_opid_index: -1
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 12:19:59.920490 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } }
I20260620 12:19:59.922134 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e06b3f42475f46e4b74cc800eac7b1cf (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 12:19:59.923018 7001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:19:59.924391 7001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Refusing update from remote peer f5e86be99fe3493c8908628297b224cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 12:19:59.925129 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:19:59.925314 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43043
I20260620 12:19:59.925611 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } }
I20260620 12:19:59.925700 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:19:59.925737 6959 master.cc:537] Master@127.0.98.61:43043 shutting down...
W20260620 12:19:59.927067 6480 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 12:19:59.930655 6959 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:19:59.930919 6959 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:19:59.931010 6959 tablet_replica.cc:330] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: stopping tablet replica
I20260620 12:19:59.935142 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.934856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.0.1:57180)
0620 12:19:59.935006 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.935066 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 12:19:59.935256 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc -> Peer e06b3f42475f46e4b74cc800eac7b1cf (127.0.98.61:43043): Couldn't send request to peer e06b3f42475f46e4b74cc800eac7b1cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43043: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 12:19:59.935326 6959 master.cc:552] Master@127.0.98.61:43043 shutdown complete.
I20260620 12:19:59.935376 6959 master_runner.cc:305] Clearing existing system tablet
I20260620 12:19:59.935676 6959 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:59.935757 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:59.937085 6959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 12:19:59.937994 7062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.938225 6959 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:19:59.938341 6959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:19:59.938367 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
uuid: "e06b3f42475f46e4b74cc800eac7b1cf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "718978327de1d7488e3eb9e1e7a0b392"
server_key_iv: "d5677c8c4e632e81c739a68c2b658db0"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 12:19:59.940424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46505
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37687,127.0.98.61:43043,127.0.98.60:46505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:19:59.940442 6959 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:19:59.942620 6959 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 12:19:59.942721 6959 log.cc:1196] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 12:19:59.942993 6959 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Deleting consensus metadata
I20260620 12:19:59.943159 6959 master_runner.cc:315] Copying system tablet from 127.0.98.62:37687
I20260620 12:19:59.944425 6959 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37687
I20260620 12:19:59.957013 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.945330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44040)
0620 12:19:59.948844 (+ 3514us) negotiation.cc:107] Waiting for socket to connect
0620 12:19:59.948859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:19:59.948974 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:19:59.949304 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:19:59.949307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:19:59.949542 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:19:59.949664 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.949671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.950843 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.950847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.951782 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:19:59.951792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.952369 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.952373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:19:59.952472 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.953132 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:19:59.953146 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:19:59.953556 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:19:59.955505 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.955508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.955511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.955749 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.956036 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:19:59.956039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:19:59.956041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:19:59.956126 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:19:59.956508 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:19:59.956512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:19:59.956602 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 12:19:59.956703 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 12:19:59.956828 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3442,"thread_start_us":94,"threads_started":1}
I20260620 12:19:59.957108 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:19:59.945433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.0.98.62:37687)
0620 12:19:59.945545 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 12:19:59.945549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:19:59.948971 (+ 3422us) server_negotiation.cc:407] Connection header received
0620 12:19:59.949112 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:19:59.949116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:19:59.949180 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:19:59.949276 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:19:59.949793 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.950721 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.951901 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:19:59.952257 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:19:59.952356 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:19:59.953693 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:19:59.953697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:19:59.953700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:19:59.953725 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:19:59.955397 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.955847 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.955852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.955854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.955935 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:19:59.956214 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:19:59.956217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:19:59.956219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:19:59.956418 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:19:59.956509 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:19:59.956764 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 12:19:59.956871 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 12:19:59.957944 6515 tablet_copy_service.cc:140] P f5e86be99fe3493c8908628297b224cc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e06b3f42475f46e4b74cc800eac7b1cf ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44040)
I20260620 12:19:59.958122 6515 tablet_copy_service.cc:161] P f5e86be99fe3493c8908628297b224cc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e06b3f42475f46e4b74cc800eac7b1cf at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44040: session id = e06b3f42475f46e4b74cc800eac7b1cf-00000000000000000000000000000000
I20260620 12:19:59.959771 6515 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 12:19:59.961452 6959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 12:19:59.963721 6959 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: tablet copy: Starting download of 0 data blocks...
I20260620 12:19:59.963900 6959 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: tablet copy: Starting download of 1 WAL segments...
I20260620 12:19:59.965502 6959 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 12:19:59.967144 6515 tablet_copy_service.cc:342] P f5e86be99fe3493c8908628297b224cc: Request end of tablet copy session e06b3f42475f46e4b74cc800eac7b1cf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44040
I20260620 12:19:59.967254 6515 tablet_copy_service.cc:433] P f5e86be99fe3493c8908628297b224cc: ending tablet copy session e06b3f42475f46e4b74cc800eac7b1cf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e06b3f42475f46e4b74cc800eac7b1cf
I20260620 12:19:59.974711 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:19:59.976375 7070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:59.976637 7073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:19:59.976768 7071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:19:59.977222 6959 server_base.cc:1034] running on GCE node
I20260620 12:19:59.977363 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:19:59.977632 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:19:59.978771 6959 hybrid_clock.cc:648] HybridClock initialized: now 1781957999978760 us; error 29 us; skew 500 ppm
Jun 20 12:19:59 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957999, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:19:59.980350 6959 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 12:19:59.980782 6959 webserver.cc:460] Webserver started at http://127.0.98.61:38469/ using document root <none> and password file <none>
I20260620 12:19:59.980960 6959 fs_manager.cc:359] Metadata directory not provided
I20260620 12:19:59.981001 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:19:59.982256 6959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 12:19:59.983237 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:19:59.983512 6959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 12:19:59.983584 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
uuid: "e06b3f42475f46e4b74cc800eac7b1cf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "718978327de1d7488e3eb9e1e7a0b392"
server_key_iv: "d5677c8c4e632e81c739a68c2b658db0"
server_key_version: "encryptionkey@0"
I20260620 12:19:59.983665 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:00.029593 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:00.031944 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:00.042156 6959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43043
I20260620 12:20:00.042781 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:20:00.045361 7132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43043 every 8 connection(s)
I20260620 12:20:00.046873 7133 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:20:00.047854 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Bootstrap starting.
I20260620 12:20:00.058043 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:20:00.058360 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: Bootstrap complete.
I20260620 12:20:00.058930 7133 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } }
I20260620 12:20:00.059067 7133 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e06b3f42475f46e4b74cc800eac7b1cf, State: Initialized, Role: LEARNER
I20260620 12:20:00.059165 7133 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } }
I20260620 12:20:00.059516 7136 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } }
I20260620 12:20:00.059640 7133 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:20:00.059712 7136 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: This master's current role is: LEARNER
I20260620 12:20:00.062948 7147 catalog_manager.cc:1259] Loaded cluster ID: 24156dbf37c0475cb36282614b8402d0
I20260620 12:20:00.063088 7147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: loading cluster ID for follower catalog manager: success
I20260620 12:20:00.065313 7147 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: acquiring CA information for follower catalog manager: success
I20260620 12:20:00.066756 7147 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 12:20:00.072584 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:00.072836 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:00.072913 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:00.077060 7063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:20:00.077117 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:00.077165 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:00.077203 7063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:20:00.077230 7063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:20:00.082563 7063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37687,127.0.98.61:43043,127.0.98.60:46505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:00.083846 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:00.085038 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:00.093730 7152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:00.093815 7155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:00.093794 7063 server_base.cc:1034] running on GCE node
W20260620 12:20:00.093731 7153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:00.094419 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:00.095481 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:00.096645 7063 hybrid_clock.cc:648] HybridClock initialized: now 1781958000096624 us; error 55 us; skew 500 ppm
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:00.099471 7063 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 12:20:00.100350 7063 webserver.cc:460] Webserver started at http://127.0.98.60:45183/ using document root <none> and password file <none>
I20260620 12:20:00.100950 7063 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:00.101048 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:00.101297 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:00.103293 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data/instance:
uuid: "1112e7032e0645afb6074ac15f732b76"
format_stamp: "Formatted at 2026-06-20 12:20:00 on dist-test-slave-9b3m"
server_key: "39e63a538e0c36ced9892052cd289bde"
server_key_iv: "9c199ab6610b687ab0aeea9580b7b919"
server_key_version: "encryptionkey@0"
I20260620 12:20:00.103875 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal/instance:
uuid: "1112e7032e0645afb6074ac15f732b76"
format_stamp: "Formatted at 2026-06-20 12:20:00 on dist-test-slave-9b3m"
server_key: "39e63a538e0c36ced9892052cd289bde"
server_key_iv: "9c199ab6610b687ab0aeea9580b7b919"
server_key_version: "encryptionkey@0"
I20260620 12:20:00.107671 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 12:20:00.110293 7162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.111557 7063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:00.111675 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
uuid: "1112e7032e0645afb6074ac15f732b76"
format_stamp: "Formatted at 2026-06-20 12:20:00 on dist-test-slave-9b3m"
server_key: "39e63a538e0c36ced9892052cd289bde"
server_key_iv: "9c199ab6610b687ab0aeea9580b7b919"
server_key_version: "encryptionkey@0"
I20260620 12:20:00.111776 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:00.129236 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:00.132624 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:00.132835 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:00.140614 7063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46505
I20260620 12:20:00.140996 7214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46505 every 8 connection(s)
I20260620 12:20:00.141685 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:20:00.144554 7215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:00.147622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 7063
I20260620 12:20:00.148420 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 12:20:00.161428 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.150503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.0.98.62:37687)
0620 12:20:00.150620 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.150625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.150812 (+ 187us) server_negotiation.cc:407] Connection header received
0620 12:20:00.151110 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.151113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.151174 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.151267 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.152056 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.152589 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.153649 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.153861 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.153992 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.157749 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.157757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.157759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.157781 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.159390 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.159996 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.160002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.160007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.160100 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.160373 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.160375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.160376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.160537 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.160641 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.161117 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.161233 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:20:00.162369 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.150324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44054)
0620 12:20:00.150686 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.150722 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.150931 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.151314 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.151319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.151677 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.151916 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.151928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.152715 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.152725 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.153527 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.153536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.154036 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.154040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.154181 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.154953 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.154973 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.157612 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.159553 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.159559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.159566 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.159902 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.160191 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.160194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.160195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.160283 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.160677 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.160684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.160840 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.161026 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.161389 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":2,"spinlock_wait_cycles":3968,"thread_start_us":94,"threads_started":1}
I20260620 12:20:00.162369 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.149169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.0.98.60:46505)
0620 12:20:00.149600 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.149614 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.149649 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:20:00.150540 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.150561 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.150944 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.151171 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.152008 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.152849 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.153732 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.154101 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.154291 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.157185 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.157191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.157203 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.157228 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.159112 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.159575 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.159582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.159588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.159750 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.160062 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.160065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.160066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.160548 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.161465 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.161562 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.161682 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":295,"spinlock_wait_cycles":1792,"thread_start_us":143,"threads_started":1}
I20260620 12:20:00.163892 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:20:00.172680 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.164425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.0.1:57184)
0620 12:20:00.164814 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.164826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.164892 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.165174 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.165176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.165307 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.165820 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.165827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.166761 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.166764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.167331 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.167339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.167770 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.167772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.167840 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.168279 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.168289 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.169595 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.171344 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.171347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.171349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.171568 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.171830 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.171832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.171833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.171911 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.172245 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.172247 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.172338 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.172434 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.172537 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314}
I20260620 12:20:00.172684 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.164535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.0.98.61:43043)
0620 12:20:00.164807 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.164823 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.164862 (+ 39us) server_negotiation.cc:407] Connection header received
0620 12:20:00.164955 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.164957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.164997 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.165126 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.165955 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.166656 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.167445 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.167676 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.167759 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.169699 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.169702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.169704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.169725 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.171243 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.171667 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.171670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.171671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.171748 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.171992 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.171994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.171995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.172162 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.172237 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.172467 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.172564 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":128,"threads_started":1}
I20260620 12:20:00.173539 7215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 12:20:00.179997 6686 heartbeater.cc:503] Master 127.0.98.62:37687 was elected leader, sending a full tablet report...
I20260620 12:20:00.181355 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.174305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.0.98.60:46505)
0620 12:20:00.174429 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.174433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.174470 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:20:00.174496 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.174498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.174529 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.174583 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.175026 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.175525 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.176244 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.176482 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.176572 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.178559 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.178562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.178564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.178583 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.179777 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.180208 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.180214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.180216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.180291 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.180559 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.180562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.180564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.180724 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.180825 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.181086 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.181225 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":55}
I20260620 12:20:00.181355 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.174184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.0.1:57496)
0620 12:20:00.174330 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.174346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.174424 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.174621 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.174624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.174783 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.174873 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.174880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.175647 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.175650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.176138 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.176145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.176583 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.176585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.176641 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.177199 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.177210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.178443 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.179885 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.179888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.179889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.180100 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.180380 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.180386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.180388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.180467 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.180814 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.180817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.180898 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.181023 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.181174 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 12:20:00.184621 7215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Bootstrap starting.
I20260620 12:20:00.186909 7215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:00.187618 7215 log.cc:824] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:00.189563 7215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: No bootstrap required, opened a new log
I20260620 12:20:00.192448 7215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } }
I20260620 12:20:00.192690 7215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:00.192771 7215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1112e7032e0645afb6074ac15f732b76, State: Initialized, Role: FOLLOWER
I20260620 12:20:00.193248 7215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } }
I20260620 12:20:00.193748 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } } }
I20260620 12:20:00.193912 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:20:00.194387 7215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 12:20:00.199286 7232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:20:00.199379 7232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:20:00.202472 7063 master_runner.cc:428] Detected that this master 1112e7032e0645afb6074ac15f732b76 is joining an existing cluster
I20260620 12:20:00.202541 7063 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46505
I20260620 12:20:00.214489 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.204982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.0.98.62:37687)
0620 12:20:00.205158 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.205163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.205771 (+ 608us) server_negotiation.cc:407] Connection header received
0620 12:20:00.205981 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.205985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.206036 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.206089 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.206816 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.207647 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.209426 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.209783 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.209876 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.211382 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.211385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.211387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.211409 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.212779 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.213293 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.213299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.213300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.213410 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.213689 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.213692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.213694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.213873 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.213973 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.214233 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.214353 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 12:20:00.214506 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.204937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44070)
0620 12:20:00.205262 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.205277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.205849 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.206231 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.206234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.206524 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.206671 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.206679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.207790 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.207794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.209290 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.209301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.210073 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.210077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.210174 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.210732 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.210747 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.211236 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.212885 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.212888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.212891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.213182 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.213494 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.213498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.213500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.213591 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.213974 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.213977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.214072 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.214173 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.214313 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":22,"thread_start_us":150,"threads_started":1}
I20260620 12:20:00.216017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 6557
I20260620 12:20:00.222630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35671
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37687
--webserver_port=45193
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043,127.0.98.60:46505 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781957998, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 12:20:00.234692 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.205798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.0.1:57508)
0620 12:20:00.206355 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.206370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.206476 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.206863 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.206867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.207069 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.207176 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.207184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.208117 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.208119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.209660 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.209670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.210266 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.210269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.210481 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.229608 (+ 19127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.229624 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.230562 (+ 938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.233257 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.233260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.233262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.233486 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.233723 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.233725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.233727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.233795 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.234187 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.234190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.234262 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.234350 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.234488 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":83,"spinlock_wait_cycles":6528,"thread_start_us":93,"threads_started":1}
I20260620 12:20:00.235283 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.206102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.98.60:46505)
0620 12:20:00.206210 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.206215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.206425 (+ 210us) server_negotiation.cc:407] Connection header received
0620 12:20:00.206591 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.206596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.206637 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.206693 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.207293 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.207994 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.209758 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.210058 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.210132 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.230707 (+ 20575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.230710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.230713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.230734 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.233151 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.233576 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.233579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.233581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.233645 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.233872 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.233875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.233876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.234090 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.234204 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.235018 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.235124 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260620 12:20:00.236248 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.205506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.0.98.61:43043)
0620 12:20:00.205633 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.205640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.205901 (+ 261us) server_negotiation.cc:407] Connection header received
0620 12:20:00.206098 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.206101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.206160 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.206221 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.207049 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.207879 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.209682 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.210082 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.210146 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.231916 (+ 21770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.231923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.231930 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.231970 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.233774 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.234404 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.234410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.234413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.234513 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.235072 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.235077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.235080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.235308 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.235376 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.235972 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.236101 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:20:00.236919 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.205426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.0.1:57194)
0620 12:20:00.205811 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.205826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.205970 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.206406 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.206412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.206733 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.206899 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.206910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.208022 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.208027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.209503 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.209513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.210266 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.210269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.210476 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.230369 (+ 19893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.230388 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.231697 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.233937 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.233942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.233946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.234274 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.234625 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.234628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.234630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.234711 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.235685 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.235688 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.235790 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.236593 (+ 803us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.236730 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":141,"spinlock_wait_cycles":2816,"thread_start_us":94,"threads_started":1}
I20260620 12:20:00.239787 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.218388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.0.1:57514)
0620 12:20:00.218573 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.218588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.218661 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.221512 (+ 2851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.221515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.221706 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.221807 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.221813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.222975 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.222978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.223831 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.223840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.224457 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.224462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.224585 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.225129 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.225149 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.226744 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.238193 (+ 11449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.238197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.238199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.238473 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.238761 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.238764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.238766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.238864 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.239261 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.239264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.239367 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.239479 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.239629 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 12:20:00.239805 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.218457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.0.98.60:46505)
0620 12:20:00.221231 (+ 2774us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.221238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.221252 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:00.221316 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.221319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.221367 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.221912 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.222034 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.222816 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.223984 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.224335 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.224429 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.226928 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.226935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.226939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.226974 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.237961 (+ 10987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.238578 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.238582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.238584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.238662 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.238969 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.238972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.238975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.239163 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.239252 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.239539 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.239649 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2700,"thread_start_us":108,"threads_started":1}
I20260620 12:20:00.240729 6495 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46505: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1112e7032e0645afb6074ac15f732b76"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46505
}
attrs {
promote: true
}
}
dest_uuid: "f5e86be99fe3493c8908628297b224cc"
cas_config_opid_index: 5
I20260620 12:20:00.241406 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } }
I20260620 12:20:00.243003 7189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:00.242931 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1112e7032e0645afb6074ac15f732b76 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.244210 7189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Refusing update from remote peer f5e86be99fe3493c8908628297b224cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 12:20:00.244329 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46505
I20260620 12:20:00.244513 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.244607 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:00.244812 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:00.245529 7063 master.cc:537] Master@127.0.98.60:46505 shutting down...
W20260620 12:20:00.248173 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc -> Peer 1112e7032e0645afb6074ac15f732b76 (127.0.98.60:46505): Couldn't send request to peer 1112e7032e0645afb6074ac15f732b76. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 12:20:00.250540 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.242615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.0.1:57202)
0620 12:20:00.242718 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.242734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.242806 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.246430 (+ 3624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.246436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.246458 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:00.246547 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.246554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.247606 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.247610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.248998 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.249010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.250142 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.250147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.250232 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.250271 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.250336 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.250409 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 12:20:00.251089 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.242819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.0.98.61:43043)
0620 12:20:00.246024 (+ 3205us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.246029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.246044 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:20:00.246094 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.246097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.246166 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.246266 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:00.246673 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.247475 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.249164 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.249811 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.249888 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.250046 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.250779 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.250851 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3111}
I20260620 12:20:00.251590 7063 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:00.251833 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 LEARNER]: Refusing update from remote peer f5e86be99fe3493c8908628297b224cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 12:20:00.251878 7063 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:00.251969 7063 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: stopping tablet replica
I20260620 12:20:00.252305 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 12:20:00.254227 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1112e7032e0645afb6074ac15f732b76 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.256405 7136 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e86be99fe3493c8908628297b224cc. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.256516 7136 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: This master's current role is: LEARNER
I20260620 12:20:00.256603 7136 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: true } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.256664 7136 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: This master's current role is: LEARNER
I20260620 12:20:00.257766 7063 master.cc:552] Master@127.0.98.60:46505 shutdown complete.
I20260620 12:20:00.257815 7063 master_runner.cc:305] Clearing existing system tablet
I20260620 12:20:00.258054 7063 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:00.258106 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:00.258409 7054 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: attempting to promote NON_VOTER e06b3f42475f46e4b74cc800eac7b1cf to VOTER
I20260620 12:20:00.258852 7054 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } }
I20260620 12:20:00.259333 7063 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 12:20:00.260061 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.259815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.0.1:57530)
0620 12:20:00.259931 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.259987 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 12:20:00.260274 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 LEARNER]: Refusing update from remote peer f5e86be99fe3493c8908628297b224cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 12:20:00.260667 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 12:20:00.261229 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.260889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.0.1:57544)
0620 12:20:00.261106 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.261153 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260620 12:20:00.261355 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc -> Peer 1112e7032e0645afb6074ac15f732b76 (127.0.98.60:46505): Couldn't send request to peer 1112e7032e0645afb6074ac15f732b76. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 12:20:00.262256 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e06b3f42475f46e4b74cc800eac7b1cf (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.263702 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.263796 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:00.263774 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.263911 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:00.264353 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e06b3f42475f46e4b74cc800eac7b1cf (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.265712 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.266093 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:20:00.266180 7246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.266428 7063 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:00.266566 7063 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260620 12:20:00.266602 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
uuid: "1112e7032e0645afb6074ac15f732b76"
format_stamp: "Formatted at 2026-06-20 12:20:00 on dist-test-slave-9b3m"
server_key: "39e63a538e0c36ced9892052cd289bde"
server_key_iv: "9c199ab6610b687ab0aeea9580b7b919"
server_key_version: "encryptionkey@0"
I20260620 12:20:00.268651 7063 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:00.271052 7063 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 12:20:00.271139 7063 log.cc:1196] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 12:20:00.271376 7063 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Deleting consensus metadata
I20260620 12:20:00.271560 7063 master_runner.cc:315] Copying system tablet from 127.0.98.62:37687
I20260620 12:20:00.272640 7063 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37687
I20260620 12:20:00.286019 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.277487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.0.98.62:37687)
0620 12:20:00.277615 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.277621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.277781 (+ 160us) server_negotiation.cc:407] Connection header received
0620 12:20:00.277917 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.277928 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.277981 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.278068 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.278584 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.279400 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.280617 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.280968 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.281096 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.282373 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.282377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.282380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.282411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.284214 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.284669 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.284673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.284674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.284760 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.285150 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.285153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.285155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.285368 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.285467 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.285744 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.285856 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:20:00.286026 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.277404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44078)
0620 12:20:00.277670 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.277686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.277804 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.278100 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.278104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.278336 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.278466 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.278473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.279531 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.279537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.280490 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.280501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.281108 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.281112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.281205 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.281797 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.281811 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.282226 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.284324 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.284327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.284329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.284569 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.284853 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.284856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.284858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.285010 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.285468 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.285471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.285569 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.285674 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.285810 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260620 12:20:00.286789 6515 tablet_copy_service.cc:140] P f5e86be99fe3493c8908628297b224cc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1112e7032e0645afb6074ac15f732b76 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44078)
I20260620 12:20:00.286886 6515 tablet_copy_service.cc:161] P f5e86be99fe3493c8908628297b224cc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1112e7032e0645afb6074ac15f732b76 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44078: session id = 1112e7032e0645afb6074ac15f732b76-00000000000000000000000000000000
I20260620 12:20:00.287941 6515 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 12:20:00.289278 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 12:20:00.291653 7063 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: tablet copy: Starting download of 0 data blocks...
I20260620 12:20:00.291822 7063 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: tablet copy: Starting download of 1 WAL segments...
I20260620 12:20:00.293277 7063 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 12:20:00.295061 6515 tablet_copy_service.cc:342] P f5e86be99fe3493c8908628297b224cc: Request end of tablet copy session 1112e7032e0645afb6074ac15f732b76-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44078
I20260620 12:20:00.295148 6515 tablet_copy_service.cc:433] P f5e86be99fe3493c8908628297b224cc: ending tablet copy session 1112e7032e0645afb6074ac15f732b76-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1112e7032e0645afb6074ac15f732b76
I20260620 12:20:00.302527 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:00.304395 7253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:00.306005 7254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:00.306061 7256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:00.306381 7063 server_base.cc:1034] running on GCE node
I20260620 12:20:00.306586 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:00.306877 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:00.308032 7063 hybrid_clock.cc:648] HybridClock initialized: now 1781958000308020 us; error 29 us; skew 500 ppm
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:00.309710 7063 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 12:20:00.310241 7063 webserver.cc:460] Webserver started at http://127.0.98.60:36509/ using document root <none> and password file <none>
I20260620 12:20:00.310422 7063 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:00.310463 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:00.311816 7063 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:00.312810 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.313079 7063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:00.313149 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
uuid: "1112e7032e0645afb6074ac15f732b76"
format_stamp: "Formatted at 2026-06-20 12:20:00 on dist-test-slave-9b3m"
server_key: "39e63a538e0c36ced9892052cd289bde"
server_key_iv: "9c199ab6610b687ab0aeea9580b7b919"
server_key_version: "encryptionkey@0"
I20260620 12:20:00.313220 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:00.322904 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:00.325119 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:00.329895 7063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46505
I20260620 12:20:00.330600 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:20:00.332427 7316 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:20:00.333534 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Bootstrap starting.
I20260620 12:20:00.337188 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46505 every 8 connection(s)
I20260620 12:20:00.344771 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:20:00.345134 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Bootstrap complete.
I20260620 12:20:00.345724 7316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } }
I20260620 12:20:00.345889 7316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1112e7032e0645afb6074ac15f732b76, State: Initialized, Role: LEARNER
I20260620 12:20:00.346025 7316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } }
I20260620 12:20:00.346386 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.346496 7316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:20:00.346596 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: This master's current role is: LEARNER
I20260620 12:20:00.350299 7330 catalog_manager.cc:1259] Loaded cluster ID: 24156dbf37c0475cb36282614b8402d0
I20260620 12:20:00.350358 7330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: loading cluster ID for follower catalog manager: success
I20260620 12:20:00.352401 7330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: acquiring CA information for follower catalog manager: success
I20260620 12:20:00.353785 7330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 12:20:00.361421 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:00.361691 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:00.361764 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:00.366026 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:00.366080 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:00.366168 7242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:20:00.371608 7242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45193
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043,127.0.98.60:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:00.372637 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:00.373680 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:00.384488 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:00.384682 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:00.384467 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:00.384961 7242 server_base.cc:1034] running on GCE node
I20260620 12:20:00.385525 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:00.386240 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:00.387435 7242 hybrid_clock.cc:648] HybridClock initialized: now 1781958000387413 us; error 31 us; skew 500 ppm
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:00.390275 7242 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 12:20:00.391106 7242 webserver.cc:460] Webserver started at http://127.0.98.1:45193/ using document root <none> and password file <none>
I20260620 12:20:00.391709 7242 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:00.391784 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:00.395514 7242 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:00.397495 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.398700 7242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:00.398833 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "7ad5a9dc70444049a8340f57f8928cd7"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "fec58798cf12eb50d066dc29d6b140bb"
server_key_iv: "640dae1fa4d8c1a304ab25568753b9f4"
server_key_version: "encryptionkey@0"
I20260620 12:20:00.399385 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:00.400941 6820 heartbeater.cc:503] Master 127.0.98.62:37687 was elected leader, sending a full tablet report...
I20260620 12:20:00.414611 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:00.417522 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:00.417718 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:00.418278 7242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:00.419106 7242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:00.419178 7242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.419250 7242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:00.419291 7242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.428743 7242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35671
I20260620 12:20:00.428818 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35671 every 8 connection(s)
I20260620 12:20:00.429803 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:20:00.439774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 7242
I20260620 12:20:00.439968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 6691
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:20:00.446667 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.433491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37635 (local address 127.0.98.62:37687)
0620 12:20:00.433621 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.433626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.433741 (+ 115us) server_negotiation.cc:407] Connection header received
0620 12:20:00.434283 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.434286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.434331 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.434387 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.435916 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.436525 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.437960 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.438288 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.438414 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.442385 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.442392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.442396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.442427 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.444382 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.445130 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.445133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.445135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.445210 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.445762 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.445766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.445767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.445990 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.446071 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.446389 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.446508 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:20:00.446859 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.433943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47213 (local address 127.0.98.60:46505)
0620 12:20:00.434217 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.434223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.434237 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:00.434281 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.434284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.434332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.434416 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.435930 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.436977 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.438123 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.438394 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.438508 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.441851 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.441854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.441857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.441887 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.443921 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.444553 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.444558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.444560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.444640 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.445064 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.445069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.445071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.445311 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.445398 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.446561 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.446675 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260620 12:20:00.448112 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.431594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57089 (local address 127.0.98.61:43043)
0620 12:20:00.431739 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.431743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.433030 (+ 1287us) server_negotiation.cc:407] Connection header received
0620 12:20:00.434271 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.434274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.434350 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.434421 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.435930 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.436727 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.437795 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.438120 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.438210 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.444462 (+ 6252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.444468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.444474 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.444508 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.446276 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.446788 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.446794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.446799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.446896 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.447222 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.447226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.447228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.447466 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.447563 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.447867 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.447980 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 12:20:00.448467 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.433355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:37635)
0620 12:20:00.433640 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.433656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.434056 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.434628 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.434631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.435144 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.435771 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.435788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.436675 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.436678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.437832 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.437841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.438435 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.438441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.438562 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.439247 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.439263 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.442238 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.444504 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.444507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.444509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.445001 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.445321 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.445324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.445560 (+ 236us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.445650 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.446125 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.446128 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.446223 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.447319 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.447451 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":47,"spinlock_wait_cycles":3840,"thread_start_us":104,"threads_started":1}
I20260620 12:20:00.448604 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.431997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.98.1:57089)
0620 12:20:00.432866 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.432910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.434039 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.434617 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.434626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.435153 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.435762 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.435779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.436855 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.436858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.437663 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.437671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.438367 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.438375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.438566 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.439301 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.439319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.444305 (+ 4986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.446404 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.446410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.446415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.446680 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.447003 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.447008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.447013 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.447097 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.447607 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.447610 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.447696 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.448250 (+ 554us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.448378 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":157,"spinlock_wait_cycles":17280,"thread_start_us":527,"threads_started":1}
I20260620 12:20:00.448622 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.433736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.98.1:47213)
0620 12:20:00.433964 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.433977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.434061 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.434617 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.434626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.435153 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.435762 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.435779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.437146 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.437149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.438016 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.438022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.438633 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.438636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.438736 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.439476 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.439493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.441651 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.444056 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.444062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.444077 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.444446 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.444750 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.444755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.444758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.444927 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.445465 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.445473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.445779 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.446201 (+ 422us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.447248 (+ 1047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":114,"spinlock_wait_cycles":32896,"thread_start_us":87,"threads_started":1}
I20260620 12:20:00.449965 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37687
I20260620 12:20:00.450073 7459 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46505
I20260620 12:20:00.450227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43129
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37687
--webserver_port=39617
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043,127.0.98.60:46505 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:20:00.450229 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43043
I20260620 12:20:00.450201 7460 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.450330 7461 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.450201 7459 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.450887 7460 heartbeater.cc:511] Master 127.0.98.62:37687 requested a full tablet report, sending...
I20260620 12:20:00.450887 7461 heartbeater.cc:511] Master 127.0.98.61:43043 requested a full tablet report, sending...
I20260620 12:20:00.451030 7459 heartbeater.cc:511] Master 127.0.98.60:46505 requested a full tablet report, sending...
I20260620 12:20:00.452135 6495 ts_manager.cc:194] Re-registered known tserver with Master: 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671)
I20260620 12:20:00.452400 7097 ts_manager.cc:194] Registered new tserver with Master: 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671)
I20260620 12:20:00.452400 7280 ts_manager.cc:194] Registered new tserver with Master: 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671)
I20260620 12:20:00.452847 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37635
W20260620 12:20:00.558800 7467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:00.559058 7467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:00.559126 7467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:00.563417 7467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:00.563493 7467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:00.563589 7467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:20:00.568840 7467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39617
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043,127.0.98.60:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:00.569929 7467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:00.570973 7467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:00.580281 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:00.580317 7467 server_base.cc:1034] running on GCE node
W20260620 12:20:00.580260 7472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:00.580246 7473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:00.580930 7467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:00.581605 7467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:00.582759 7467 hybrid_clock.cc:648] HybridClock initialized: now 1781958000582730 us; error 43 us; skew 500 ppm
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:00.585737 7467 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 12:20:00.586900 7467 webserver.cc:460] Webserver started at http://127.0.98.2:39617/ using document root <none> and password file <none>
I20260620 12:20:00.587463 7467 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:00.587548 7467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:00.591547 7467 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:20:00.593860 7482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.595276 7467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:20:00.595418 7467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "29f788b0584444fcaa8c693eabd2817a"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "9cf41b188c40403e2b33e64967ef35fc"
server_key_iv: "9c0ca17e40379862fdde32b52b8e5ea8"
server_key_version: "encryptionkey@0"
I20260620 12:20:00.596057 7467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:00.617473 7467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:00.620976 7467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:00.621210 7467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:00.621795 7467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:00.622622 7467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:00.622666 7467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.622733 7467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:00.622763 7467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.628396 6954 heartbeater.cc:503] Master 127.0.98.62:37687 was elected leader, sending a full tablet report...
I20260620 12:20:00.632408 7467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43129
I20260620 12:20:00.632431 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43129 every 8 connection(s)
I20260620 12:20:00.633489 7467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:20:00.636575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 7467
I20260620 12:20:00.636772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 6825
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:20:00.645790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33747
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37687
--webserver_port=38561
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043,127.0.98.60:46505 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 12:20:00.650755 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.635263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45087 (local address 127.0.98.61:43043)
0620 12:20:00.635395 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.635399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.636704 (+ 1305us) server_negotiation.cc:407] Connection header received
0620 12:20:00.638183 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.638186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.638231 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.638331 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.639897 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.640453 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.642318 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.642777 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.642872 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.646192 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.646199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.646205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.646238 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.648011 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.648777 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.648784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.648789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.648871 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.649338 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.649347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.649352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.649619 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.649723 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.650430 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.650557 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:20:00.652326 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.635638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.98.2:45087)
0620 12:20:00.636546 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.636603 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.637957 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.638509 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.638521 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.639035 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.639722 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.639743 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.640589 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.640595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.642172 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.642186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.643076 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.643082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.643304 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.644009 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.644033 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.646009 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.648141 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.648147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.648163 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.648670 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.648993 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.648998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.649004 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.649199 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.649782 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.649791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.650121 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.650945 (+ 824us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.651282 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"spinlock_wait_cycles":896,"thread_start_us":525,"threads_started":1}
I20260620 12:20:00.652698 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.637441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41027 (local address 127.0.98.60:46505)
0620 12:20:00.637572 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.637576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.642254 (+ 4678us) server_negotiation.cc:407] Connection header received
0620 12:20:00.642429 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.642433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.642499 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.642554 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.643141 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.643883 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.644934 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.645309 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.645384 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.648742 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.648746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.648749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.648783 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.650727 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.651212 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.651215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.651217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.651297 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.651598 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.651601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.651603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.651792 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.651888 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.652425 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.652550 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:20:00.652987 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.637302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.98.2:41027)
0620 12:20:00.642172 (+ 4870us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.642186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.642314 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.642693 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.642695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.642876 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.642990 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.642996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.644018 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.644021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.644824 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.644832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.645634 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.645638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.645751 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.646508 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.646524 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.648600 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.650874 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.650877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.650879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.651104 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.651399 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.651402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.651404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.651500 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.651929 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.651932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.652014 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.652104 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.652233 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4796,"thread_start_us":76,"threads_started":1}
I20260620 12:20:00.653455 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43043
I20260620 12:20:00.653683 7598 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.654042 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.641548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41357 (local address 127.0.98.62:37687)
0620 12:20:00.641680 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.641685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.641876 (+ 191us) server_negotiation.cc:407] Connection header received
0620 12:20:00.642028 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.642032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.642084 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.642160 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.642815 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.643606 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.644838 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.645187 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.645256 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.648829 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.648835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.648840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.648865 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.650946 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.651731 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.651737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.651740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.651831 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.652200 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.652207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.652210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.652451 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.652533 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.653670 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.653804 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:20:00.654241 7598 heartbeater.cc:511] Master 127.0.98.61:43043 requested a full tablet report, sending...
I20260620 12:20:00.654647 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.636966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:41357)
0620 12:20:00.641739 (+ 4773us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.641750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.641918 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.642307 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.642310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.642545 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.642690 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.642696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.643733 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.643736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.644709 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.644719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.645413 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.645416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.645526 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.646294 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.646332 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.648678 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.651306 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.651312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.651317 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.651600 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.651973 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.651978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.651983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.652082 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.653010 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.653015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.653392 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.654238 (+ 846us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.654407 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4708,"thread_start_us":101,"threads_started":1}
I20260620 12:20:00.655440 7596 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46505
I20260620 12:20:00.655532 7596 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.655650 7097 ts_manager.cc:194] Registered new tserver with Master: 29f788b0584444fcaa8c693eabd2817a (127.0.98.2:43129)
I20260620 12:20:00.655725 7596 heartbeater.cc:511] Master 127.0.98.60:46505 requested a full tablet report, sending...
I20260620 12:20:00.655885 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37687
I20260620 12:20:00.655967 7597 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.656119 7597 heartbeater.cc:511] Master 127.0.98.62:37687 requested a full tablet report, sending...
I20260620 12:20:00.656471 7280 ts_manager.cc:194] Registered new tserver with Master: 29f788b0584444fcaa8c693eabd2817a (127.0.98.2:43129)
I20260620 12:20:00.656985 6495 ts_manager.cc:194] Re-registered known tserver with Master: 29f788b0584444fcaa8c693eabd2817a (127.0.98.2:43129)
I20260620 12:20:00.657826 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41357
W20260620 12:20:00.758672 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:00.758915 7603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:00.758958 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:00.763259 7603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:00.763319 7603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:00.763383 7603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:20:00.768649 7603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38561
--tserver_master_addrs=127.0.98.62:37687,127.0.98.61:43043,127.0.98.60:46505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:00.769721 7603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:00.770712 7603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:00.780802 7609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:00.780861 7611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:00.781126 7603 server_base.cc:1034] running on GCE node
W20260620 12:20:00.780782 7608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:00.781714 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:00.781889 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.776502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.0.98.60:46505)
0620 12:20:00.776649 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.776657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.776831 (+ 174us) server_negotiation.cc:407] Connection header received
0620 12:20:00.777093 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.777100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.777167 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.777291 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:00.777610 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.778471 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.780507 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.781344 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.781462 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.781613 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.781682 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.781761 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:20:00.782032 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.776373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.0.1:57546)
0620 12:20:00.776723 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.776738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.776895 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.777327 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.777331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.777354 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:00.777479 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.777486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.778636 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.778641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.779975 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.779993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.781489 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.781493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.781573 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.781596 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.781668 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.781844 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":74,"threads_started":1}
I20260620 12:20:00.782475 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:00.782707 7290 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 12:20:00.783093 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e86be99fe3493c8908628297b224cc. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.783248 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: This master's current role is: LEARNER
I20260620 12:20:00.783628 7603 hybrid_clock.cc:648] HybridClock initialized: now 1781958000783603 us; error 45 us; skew 500 ppm
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:00.786947 7603 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 12:20:00.787817 7603 webserver.cc:460] Webserver started at http://127.0.98.3:38561/ using document root <none> and password file <none>
I20260620 12:20:00.788352 7603 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:00.788424 7603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:00.792308 7603 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:00.794312 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.795562 7603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:00.795682 7603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "c6253039999647e68e450a5d4993cfaf"
format_stamp: "Formatted at 2026-06-20 12:19:59 on dist-test-slave-9b3m"
server_key: "bebf8d250c0e66b18b292e9cf14dab75"
server_key_iv: "c2fec1b3adb7422de2f2e324730d8741"
server_key_version: "encryptionkey@0"
I20260620 12:20:00.796212 7603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:00.817516 7603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:00.820298 7603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:00.820501 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:00.821138 7603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:00.821995 7603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:00.822046 7603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.822116 7603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:00.822146 7603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:00.830816 7603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33747
I20260620 12:20:00.830835 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33747 every 8 connection(s)
I20260620 12:20:00.831791 7603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:20:00.832901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 7603
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 12:20:00.848809 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.834828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.0.98.62:37687)
0620 12:20:00.834949 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.834953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.836239 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 12:20:00.836384 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.836388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.836439 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.836519 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.838841 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.839582 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.841122 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.841450 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.841523 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.843379 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.843383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.843385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.843420 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.846069 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.846568 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.846573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.846575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.846668 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.846999 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.847004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.847006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.847243 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.847347 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.847774 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.848627 (+ 853us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 12:20:00 dist-test-slave-9b3m krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 12:20:00.851506 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.833588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37675 (local address 127.0.98.61:43043)
0620 12:20:00.833733 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.833739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.835174 (+ 1435us) server_negotiation.cc:407] Connection header received
0620 12:20:00.836666 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.836670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.836718 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.836783 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.839304 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.840127 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.843027 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.843330 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.843429 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.846891 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.846896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.846901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.846932 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.848743 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.849638 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.849643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.849644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.849729 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.850137 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.850140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.850142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.850381 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.850469 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.851137 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.851283 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:20:00.851573 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.835492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.98.60:46505)
0620 12:20:00.835837 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.835841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.836617 (+ 776us) server_negotiation.cc:407] Connection header received
0620 12:20:00.837732 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.837737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.837798 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.837862 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.840651 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.841423 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.843278 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.843632 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.843718 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.846395 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.846398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.846401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.846427 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.848552 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.850214 (+ 1662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.850220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.850227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.850307 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.850635 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.850639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.850644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.850863 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.850986 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.851300 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.851411 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260620 12:20:00.853148 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.834064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.98.3:37675)
0620 12:20:00.835020 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.835077 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.836447 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.837029 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.837037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.838174 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.839114 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.839135 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.840244 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.840251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.842722 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.842731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.843594 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.843597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.843689 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.845018 (+ 1329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.845033 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.846726 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.849124 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.849132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.849151 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.849539 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.849842 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.849847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.849851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.850017 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.850542 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.850548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.850865 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.851661 (+ 796us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.852028 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":1137,"spinlock_wait_cycles":12672,"thread_start_us":599,"threads_started":1}
I20260620 12:20:00.854466 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43043
I20260620 12:20:00.854701 7735 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.855239 7735 heartbeater.cc:511] Master 127.0.98.61:43043 requested a full tablet report, sending...
I20260620 12:20:00.855492 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.835626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46127 (local address 127.0.98.62:37687)
0620 12:20:00.836252 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.836256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.836270 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:00.836712 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.836716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.836771 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.836860 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.839248 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.839958 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.841999 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.842379 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.842486 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.846699 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.846705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.846709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.846732 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.850728 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.854068 (+ 3340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.854071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.854073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.854158 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.854440 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.854444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.854445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.854701 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.854843 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.855176 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.855344 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":75,"threads_started":1}
I20260620 12:20:00.856148 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.835497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:46127)
0620 12:20:00.835750 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.835764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.836614 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.837029 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.837037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.838174 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.839114 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.839135 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.840261 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.840264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.841882 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.841893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.842886 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.842890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.843180 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.843871 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.843892 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.846410 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.853193 (+ 6783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.853196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.853198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.853935 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.854255 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.854257 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.854259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.854336 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.854899 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.854902 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.854999 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.855796 (+ 797us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.855948 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":189,"thread_start_us":80,"threads_started":1}
I20260620 12:20:00.856384 7097 ts_manager.cc:194] Registered new tserver with Master: c6253039999647e68e450a5d4993cfaf (127.0.98.3:33747)
I20260620 12:20:00.856477 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.835842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.98.3:50695)
0620 12:20:00.836064 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.836077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.836783 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.839713 (+ 2930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.839716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.839920 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:00.840029 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.840035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.843970 (+ 3935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.843974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.844829 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.844841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.847214 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.847221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.847324 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.847760 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:00.847779 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:00.849796 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:00.852872 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.852875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.852877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.853126 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.853426 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:00.853429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:00.853431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:00.854961 (+ 1530us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x38d346b8. stack: 00007f5e3fa53c78 00007f5e3fa541a9 00007f5e3f6161d7 00000000005984a6 00000000005984f9 000000000059c488 00007f5e3fa502ab 00007f5e3fa4f9da 00007f5e441280c4 00007f5e440abcbe 00007f5e440ab2c8 00007f5e440aa146 00007f5e440a6e17 00007f5e440e7832 00007f5e440e868c 00007f5e440fe0d3
0620 12:20:00.855050 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:00.855511 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:00.855514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:00.855619 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.855724 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.855871 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"spinlock_wait_cycles":2414720,"thread_start_us":89,"threads_started":1}
I20260620 12:20:00.856622 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.835182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.0.98.61:43043)
0620 12:20:00.837020 (+ 1838us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.837025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.837063 (+ 38us) server_negotiation.cc:407] Connection header received
0620 12:20:00.837107 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.837113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.837163 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.837252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.838159 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.839009 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.842218 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.842620 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.842722 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.848437 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.848441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.848444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.848477 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.854404 (+ 5927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.854935 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.854938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.854940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.855022 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.855339 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.855343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.855345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.855544 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.855656 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.856322 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.856440 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446,"thread_start_us":74,"threads_started":1}
I20260620 12:20:00.856999 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.836163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50695 (local address 127.0.98.60:46505)
0620 12:20:00.839390 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.839404 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.839416 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:20:00.839452 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.839455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.839497 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.839586 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:00.840168 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.842472 (+ 2304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.845601 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.845992 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.846100 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.850945 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:00.850949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:00.850951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:00.850973 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:00.852757 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.853242 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.853245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.853247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.853328 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:00.855157 (+ 1829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:00.855161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:00.855165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:00.855384 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:00.855460 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.856709 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.856842 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3151,"thread_start_us":79,"threads_started":1}
I20260620 12:20:00.857086 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37687
I20260620 12:20:00.857160 7734 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.857342 7734 heartbeater.cc:511] Master 127.0.98.62:37687 requested a full tablet report, sending...
I20260620 12:20:00.857563 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46505
I20260620 12:20:00.857638 7733 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:00.857803 7733 heartbeater.cc:511] Master 127.0.98.60:46505 requested a full tablet report, sending...
I20260620 12:20:00.851836 6495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:20:00.857981 6494 ts_manager.cc:194] Re-registered known tserver with Master: c6253039999647e68e450a5d4993cfaf (127.0.98.3:33747)
I20260620 12:20:00.858525 6494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46127
W20260620 12:20:00.858960 6495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:20:00.859261 7280 ts_manager.cc:194] Registered new tserver with Master: c6253039999647e68e450a5d4993cfaf (127.0.98.3:33747)
I20260620 12:20:00.861502 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: true } } }
I20260620 12:20:00.861621 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:00.864549 7054 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: attempting to promote NON_VOTER 1112e7032e0645afb6074ac15f732b76 to VOTER
I20260620 12:20:00.864837 7054 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } }
I20260620 12:20:00.866381 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Refusing update from remote peer f5e86be99fe3493c8908628297b224cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 12:20:00.866755 7753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 12:20:00.866847 7290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 LEARNER]: Refusing update from remote peer f5e86be99fe3493c8908628297b224cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 12:20:00.867918 7752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 12:20:00.868774 7753 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1112e7032e0645afb6074ac15f732b76 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:00.869539 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1112e7032e0645afb6074ac15f732b76 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:00.870362 7753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:00.870471 7753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:00.870321 7290 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1112e7032e0645afb6074ac15f732b76 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:00.870609 7753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:00.870692 7753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:00.870754 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:00.870852 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:00.872562 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:00.872689 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:20:00.872812 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5e86be99fe3493c8908628297b224cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:00.872947 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:20:00.879055 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.874095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.0.98.2:43129)
0620 12:20:00.874448 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.874458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.874486 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:20:00.874547 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.874550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.874686 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.875119 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:00.875266 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.876367 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.877732 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.878516 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.878637 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.878786 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.878839 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.878933 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":178,"threads_started":1}
I20260620 12:20:00.879045 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.873784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43129 (local address 127.0.0.1:53986)
0620 12:20:00.874029 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.874042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.874122 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.874849 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.874852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.874866 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:00.874978 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.874985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.876466 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.876470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.877624 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.877630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.878636 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.878640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.878723 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.878745 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.878808 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.878906 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":47,"thread_start_us":79,"threads_started":1}
I20260620 12:20:00.879388 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.873240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33747 (local address 127.0.0.1:51922)
0620 12:20:00.873367 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.873383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.873469 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.875260 (+ 1791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.875262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.875277 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:00.875342 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.875347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.876242 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.876248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.877439 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.877448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.878762 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.878766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.878906 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.878930 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.878981 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.879049 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260620 12:20:00.879598 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.874128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.0.98.3:33747)
0620 12:20:00.874886 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.874896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.874915 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:20:00.874984 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.874989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.875152 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.875284 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:00.875431 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.876128 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.877567 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.878621 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.879136 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.879304 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.879352 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.879438 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":146,"threads_started":1}
I20260620 12:20:00.880268 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.873460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.0.1:56546)
0620 12:20:00.873705 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:00.873719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:00.873814 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:00.874986 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:00.874989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:00.875007 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:00.875091 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.875096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.877163 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.877172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.878241 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:00.878252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.879901 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.879904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:00.879996 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.880016 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:20:00.880088 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 12:20:00.880150 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":21,"thread_start_us":132,"threads_started":1}
I20260620 12:20:00.880491 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:00.873613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.0.98.1:35671)
0620 12:20:00.874331 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:00.874343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:00.874367 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:20:00.874445 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:00.874450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:00.874665 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:00.874826 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:00.875191 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.876352 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.878399 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:00.879602 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:00.879901 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:00.880033 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:00.880113 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 12:20:00.880194 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":85,"threads_started":1}
I20260620 12:20:00.882158 7667 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca944fbe6f341b6ba54cd19513d39bd (DEFAULT_TABLE table=test-table [id=b2e2fa0cd2a3478586341152dfaf184d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:00.882161 7530 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca944fbe6f341b6ba54cd19513d39bd (DEFAULT_TABLE table=test-table [id=b2e2fa0cd2a3478586341152dfaf184d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:00.883276 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca944fbe6f341b6ba54cd19513d39bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:00.883287 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca944fbe6f341b6ba54cd19513d39bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:00.887171 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca944fbe6f341b6ba54cd19513d39bd (DEFAULT_TABLE table=test-table [id=b2e2fa0cd2a3478586341152dfaf184d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:00.888159 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca944fbe6f341b6ba54cd19513d39bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:00.892484 7767 tablet_bootstrap.cc:492] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: Bootstrap starting.
I20260620 12:20:00.893388 7768 tablet_bootstrap.cc:492] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Bootstrap starting.
I20260620 12:20:00.894724 7767 tablet_bootstrap.cc:654] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:00.895503 7768 tablet_bootstrap.cc:654] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:00.895560 7767 log.cc:824] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:00.896422 7768 log.cc:824] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:00.897596 7767 tablet_bootstrap.cc:492] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: No bootstrap required, opened a new log
I20260620 12:20:00.897770 7767 ts_tablet_manager.cc:1397] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:20:00.898304 7770 tablet_bootstrap.cc:492] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Bootstrap starting.
I20260620 12:20:00.898375 7768 tablet_bootstrap.cc:492] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: No bootstrap required, opened a new log
I20260620 12:20:00.898561 7768 ts_tablet_manager.cc:1397] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:20:00.900295 7770 tablet_bootstrap.cc:654] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:00.901101 7770 log.cc:824] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:00.902683 7767 raft_consensus.cc:348] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:00.902946 7767 raft_consensus.cc:374] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:00.903007 7767 raft_consensus.cc:729] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f788b0584444fcaa8c693eabd2817a, State: Initialized, Role: FOLLOWER
I20260620 12:20:00.903097 7770 tablet_bootstrap.cc:492] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: No bootstrap required, opened a new log
I20260620 12:20:00.902976 7768 raft_consensus.cc:348] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:00.903182 7768 raft_consensus.cc:374] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:00.903239 7768 raft_consensus.cc:729] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6253039999647e68e450a5d4993cfaf, State: Initialized, Role: FOLLOWER
I20260620 12:20:00.903293 7770 ts_tablet_manager.cc:1397] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:20:00.903548 7767 consensus_queue.cc:260] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:00.903640 7768 consensus_queue.cc:260] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:00.904286 7597 heartbeater.cc:503] Master 127.0.98.62:37687 was elected leader, sending a full tablet report...
I20260620 12:20:00.904549 7767 ts_tablet_manager.cc:1428] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 12:20:00.904953 7768 ts_tablet_manager.cc:1428] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:20:00.907948 7734 heartbeater.cc:503] Master 127.0.98.62:37687 was elected leader, sending a full tablet report...
I20260620 12:20:00.908902 7770 raft_consensus.cc:348] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:00.909186 7770 raft_consensus.cc:374] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:00.909253 7770 raft_consensus.cc:729] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad5a9dc70444049a8340f57f8928cd7, State: Initialized, Role: FOLLOWER
I20260620 12:20:00.909790 7770 consensus_queue.cc:260] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:00.910749 7770 ts_tablet_manager.cc:1428] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:20:00.911824 7460 heartbeater.cc:503] Master 127.0.98.62:37687 was elected leader, sending a full tablet report...
W20260620 12:20:00.931789 7462 tablet.cc:2369] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:20:01.063143 7774 raft_consensus.cc:482] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:01.063443 7774 raft_consensus.cc:504] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:01.064565 7774 leader_election.cc:288] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671), 29f788b0584444fcaa8c693eabd2817a (127.0.98.2:43129)
I20260620 12:20:01.068667 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.065197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43129 (local address 127.0.98.3:42145)
0620 12:20:01.065330 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:01.065346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:01.065417 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:01.065689 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:01.065711 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:01.065730 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:01.065832 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:01.065838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.066696 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.066698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:01.067469 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:01.067477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.068203 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.068207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:01.068308 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.068338 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 12:20:01.068410 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 12:20:01.068487 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 12:20:01.068692 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.065337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42145 (local address 127.0.98.2:43129)
0620 12:20:01.065465 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:01.065469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:01.065483 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:01.065536 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:01.065538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:01.065586 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:01.065673 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:01.065973 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.066581 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.067597 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.068081 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.068173 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.068276 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:01.068467 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 12:20:01.068559 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:20:01.068779 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.064996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33303 (local address 127.0.98.1:35671)
0620 12:20:01.065159 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:01.065165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:01.065184 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:20:01.065246 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:01.065252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:01.065305 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:01.065427 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:01.065756 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.066335 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.067718 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.068102 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.068236 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.068298 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:01.068583 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 12:20:01.068653 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:20:01.068794 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.064887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.98.3:33303)
0620 12:20:01.065018 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:01.065034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:01.065159 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:01.065465 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:01.065468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:01.065506 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:01.065638 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:01.065647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.066481 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.066484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:01.067581 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:01.067592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.068262 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.068266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:01.068376 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.068451 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 12:20:01.068534 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 12:20:01.068598 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 12:20:01.069409 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca944fbe6f341b6ba54cd19513d39bd" candidate_uuid: "c6253039999647e68e450a5d4993cfaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad5a9dc70444049a8340f57f8928cd7" is_pre_election: true
I20260620 12:20:01.069409 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca944fbe6f341b6ba54cd19513d39bd" candidate_uuid: "c6253039999647e68e450a5d4993cfaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f788b0584444fcaa8c693eabd2817a" is_pre_election: true
I20260620 12:20:01.069712 7413 raft_consensus.cc:2454] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6253039999647e68e450a5d4993cfaf in term 0.
I20260620 12:20:01.069710 7550 raft_consensus.cc:2454] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6253039999647e68e450a5d4993cfaf in term 0.
I20260620 12:20:01.070189 7622 leader_election.cc:302] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f788b0584444fcaa8c693eabd2817a, c6253039999647e68e450a5d4993cfaf; no voters:
I20260620 12:20:01.070488 7774 raft_consensus.cc:2781] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:20:01.070582 7774 raft_consensus.cc:482] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:01.070636 7774 raft_consensus.cc:3037] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:01.071702 7774 raft_consensus.cc:504] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:01.072024 7774 leader_election.cc:288] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [CANDIDATE]: Term 1 election: Requested vote from peers 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671), 29f788b0584444fcaa8c693eabd2817a (127.0.98.2:43129)
I20260620 12:20:01.072402 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca944fbe6f341b6ba54cd19513d39bd" candidate_uuid: "c6253039999647e68e450a5d4993cfaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad5a9dc70444049a8340f57f8928cd7"
I20260620 12:20:01.072512 7413 raft_consensus.cc:3037] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:01.072515 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca944fbe6f341b6ba54cd19513d39bd" candidate_uuid: "c6253039999647e68e450a5d4993cfaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f788b0584444fcaa8c693eabd2817a"
I20260620 12:20:01.072678 7550 raft_consensus.cc:3037] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:01.073586 7413 raft_consensus.cc:2454] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6253039999647e68e450a5d4993cfaf in term 1.
I20260620 12:20:01.073917 7622 leader_election.cc:302] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ad5a9dc70444049a8340f57f8928cd7, c6253039999647e68e450a5d4993cfaf; no voters:
I20260620 12:20:01.073976 7550 raft_consensus.cc:2454] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6253039999647e68e450a5d4993cfaf in term 1.
I20260620 12:20:01.074119 7774 raft_consensus.cc:2781] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:01.074347 7774 raft_consensus.cc:686] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 1 LEADER]: Becoming Leader. State: Replica: c6253039999647e68e450a5d4993cfaf, State: Running, Role: LEADER
I20260620 12:20:01.074900 7774 consensus_queue.cc:237] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } }
I20260620 12:20:01.078711 6495 catalog_manager.cc:5495] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf reported cstate change: term changed from 0 to 1, leader changed from <none> to c6253039999647e68e450a5d4993cfaf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c6253039999647e68e450a5d4993cfaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } health_report { overall_health: UNKNOWN } } }
W20260620 12:20:01.083580 7736 tablet.cc:2369] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:20:01.116410 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.112565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.0.98.3:33747)
0620 12:20:01.112703 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:01.112708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:01.112722 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:01.112833 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:01.112835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:01.112896 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:01.113008 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:20:01.113269 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.113828 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.114841 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.115158 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.115241 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.115492 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:20:01.115934 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:20:01.116029 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:01.116172 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 12:20:01.116222 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:20:01.122148 7413 raft_consensus.cc:1261] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 1 FOLLOWER]: Refusing update from remote peer c6253039999647e68e450a5d4993cfaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:01.122148 7550 raft_consensus.cc:1261] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 1 FOLLOWER]: Refusing update from remote peer c6253039999647e68e450a5d4993cfaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:01.122954 7774 consensus_queue.cc:1035] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:01.123155 7776 consensus_queue.cc:1035] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:01.132236 7781 mvcc.cc:205] Tried to move back new op lower bound from 7298899972589109248 to 7298899972405506048. Current Snapshot: MvccSnapshot[applied={T|T < 7298899972589109248}]
I20260620 12:20:01.133729 7780 mvcc.cc:205] Tried to move back new op lower bound from 7298899972589109248 to 7298899972405506048. Current Snapshot: MvccSnapshot[applied={T|T < 7298899972589109248}]
I20260620 12:20:01.135190 7779 mvcc.cc:205] Tried to move back new op lower bound from 7298899972589109248 to 7298899972405506048. Current Snapshot: MvccSnapshot[applied={T|T < 7298899972589109248}]
I20260620 12:20:01.143956 6495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:01.144151 6495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:01.148423 6495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5e86be99fe3493c8908628297b224cc: Sending DeleteTablet for 3 replicas of tablet 1ca944fbe6f341b6ba54cd19513d39bd
I20260620 12:20:01.149444 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca944fbe6f341b6ba54cd19513d39bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56546
I20260620 12:20:01.149977 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca944fbe6f341b6ba54cd19513d39bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51922
I20260620 12:20:01.150023 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca944fbe6f341b6ba54cd19513d39bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53986
I20260620 12:20:01.150633 7792 tablet_replica.cc:330] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: stopping tablet replica
I20260620 12:20:01.150825 7791 tablet_replica.cc:330] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: stopping tablet replica
I20260620 12:20:01.151129 7791 raft_consensus.cc:2229] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:20:01.151564 7791 raft_consensus.cc:2258] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:01.152321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 6463
I20260620 12:20:01.152480 7789 tablet_replica.cc:330] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: stopping tablet replica
I20260620 12:20:01.153702 7789 raft_consensus.cc:2229] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:01.154045 7789 raft_consensus.cc:2258] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:01.155397 7792 raft_consensus.cc:2229] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:01.155702 7792 raft_consensus.cc:2258] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:01.157932 7789 ts_tablet_manager.cc:1905] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:01.158624 7792 ts_tablet_manager.cc:1905] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:01.162283 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.150868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.0.98.61:43043)
0620 12:20:01.150982 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:01.150988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:01.151002 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:01.151042 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:01.151044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:01.151091 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:01.151166 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:01.152398 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.153601 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.155197 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.155534 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.155635 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.157652 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:01.157658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:01.157660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:01.157688 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:01.159584 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:01.160544 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:01.160550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:01.160552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:01.160643 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:01.161225 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:01.161229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:01.161231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:01.161487 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:01.161599 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:01.161989 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 12:20:01.162127 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 12:20:01.162475 7791 ts_tablet_manager.cc:1905] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:01.165428 7791 ts_tablet_manager.cc:1918] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:01.165527 7791 log.cc:1196] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/1ca944fbe6f341b6ba54cd19513d39bd
I20260620 12:20:01.165858 7791 ts_tablet_manager.cc:1939] T 1ca944fbe6f341b6ba54cd19513d39bd P c6253039999647e68e450a5d4993cfaf: Deleting consensus metadata
I20260620 12:20:01.167356 7792 ts_tablet_manager.cc:1918] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:01.167469 7792 log.cc:1196] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/1ca944fbe6f341b6ba54cd19513d39bd
I20260620 12:20:01.167793 7792 ts_tablet_manager.cc:1939] T 1ca944fbe6f341b6ba54cd19513d39bd P 29f788b0584444fcaa8c693eabd2817a: Deleting consensus metadata
I20260620 12:20:01.168893 7789 ts_tablet_manager.cc:1918] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:01.168994 7789 log.cc:1196] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/1ca944fbe6f341b6ba54cd19513d39bd
W20260620 12:20:01.169114 7485 connection.cc:441] server connection from 127.0.0.1:53986 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53986 (request call id 1) could send its response
I20260620 12:20:01.169319 7789 ts_tablet_manager.cc:1939] T 1ca944fbe6f341b6ba54cd19513d39bd P 7ad5a9dc70444049a8340f57f8928cd7: Deleting consensus metadata
W20260620 12:20:01.170298 7622 connection.cc:441] server connection from 127.0.0.1:51922 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51922 (request call id 1) could send its response
W20260620 12:20:01.170532 7347 connection.cc:441] server connection from 127.0.0.1:56546 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56546 (request call id 1) could send its response
I20260620 12:20:01.174386 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.150923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.98.60:46505)
0620 12:20:01.151314 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:01.151318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:01.164081 (+ 12763us) server_negotiation.cc:407] Connection header received
0620 12:20:01.164274 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:01.164277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:01.164353 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:01.164412 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:01.165366 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.166148 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.167439 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.167838 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.167928 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.170097 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:01.170101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:01.170103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:01.170133 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:01.172254 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:01.173016 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:01.173020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:01.173022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:01.173138 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:01.173442 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:01.173445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:01.173447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:01.173655 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:01.173768 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:01.174086 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 12:20:01.174202 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260620 12:20:01.175984 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.165927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.0.98.61:43043)
0620 12:20:01.166047 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:01.166050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:01.167227 (+ 1177us) server_negotiation.cc:407] Connection header received
0620 12:20:01.167407 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:01.167412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:01.167467 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:01.167526 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:01.168497 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.169266 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.170416 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.170726 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.170805 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.172334 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:01.172337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:01.172340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:01.172364 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:01.174014 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:01.174472 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:01.174476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:01.174478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:01.174553 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:01.175096 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:01.175099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:01.175101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:01.175301 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:01.175404 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:01.175737 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 12:20:01.175845 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:20:01.179338 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:01.166523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.0.98.60:46505)
0620 12:20:01.166653 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:01.166656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:01.169700 (+ 3044us) server_negotiation.cc:407] Connection header received
0620 12:20:01.170951 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:01.170955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:01.171007 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:01.171102 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:01.172062 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.172858 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.174023 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:01.174319 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:01.174388 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:01.175915 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:01.175922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:01.175927 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:01.175950 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:01.177791 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:01.178236 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:01.178240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:01.178242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:01.178321 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:01.178588 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:01.178591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:01.178593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:01.178781 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:01.178900 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:01.179087 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 12:20:01.179190 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:20:02.135465 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.134842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:57969)
0620 12:20:02.135182 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.135323 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":87,"threads_started":1}
W20260620 12:20:02.136047 7597 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:02.136356 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.135842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:37987)
0620 12:20:02.136196 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.136274 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":62,"threads_started":1}
W20260620 12:20:02.136890 7460 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:02.137530 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.136990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:48289)
0620 12:20:02.137336 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.137430 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
W20260620 12:20:02.137784 7734 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:02.649477 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5e86be99fe3493c8908628297b224cc)
I20260620 12:20:02.649519 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5e86be99fe3493c8908628297b224cc)
I20260620 12:20:02.649746 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } }
I20260620 12:20:02.649811 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } }
I20260620 12:20:02.650790 7804 leader_election.cc:288] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687), 1112e7032e0645afb6074ac15f732b76 (127.0.98.60:46505)
I20260620 12:20:02.650832 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687), e06b3f42475f46e4b74cc800eac7b1cf (127.0.98.61:43043)
I20260620 12:20:02.651752 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.650920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44492)
0620 12:20:02.651554 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.651638 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":85,"threads_started":1}
I20260620 12:20:02.651757 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.650917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44482)
0620 12:20:02.651566 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.651646 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":82,"threads_started":1}
W20260620 12:20:02.651928 7082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
W20260620 12:20:02.651928 7265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:02.652870 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.652624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44502)
0620 12:20:02.652758 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.652807 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 12:20:02.652870 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.652624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44504)
0620 12:20:02.652758 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.652807 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 12:20:02.653008 7082 leader_election.cc:334] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
W20260620 12:20:02.653110 7265 leader_election.cc:334] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:02.655821 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.651305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46505 (local address 127.0.0.1:57566)
0620 12:20:02.651631 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.651647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:02.651759 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:02.652176 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:02.652183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:02.652209 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:02.652312 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:02.652321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:02.653476 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:02.653479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:02.654661 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:02.654673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:02.655436 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:02.655438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:02.655576 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:02.655597 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:20:02.655665 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 12:20:02.655722 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":33,"thread_start_us":58,"threads_started":1}
I20260620 12:20:02.655912 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.651504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.0.98.60:46505)
0620 12:20:02.651706 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:02.651711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:02.651728 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:20:02.651905 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:02.651909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:02.651964 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:02.652043 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:02.652458 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:02.653361 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:02.654807 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:02.655318 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:02.655426 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:02.655500 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:02.655707 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 12:20:02.655779 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":66,"threads_started":1}
I20260620 12:20:02.656457 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.651237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43043 (local address 127.0.0.1:57234)
0620 12:20:02.651994 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.652087 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:02.652196 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:02.652760 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:02.652764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:02.652781 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:02.652858 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:02.652864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:02.653868 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:02.653872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:02.655007 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:02.655017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:02.656058 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:02.656063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:02.656167 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:02.656197 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 12:20:02.656285 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 12:20:02.656368 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":101,"threads_started":1}
I20260620 12:20:02.656515 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.651381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.0.98.61:43043)
0620 12:20:02.652534 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:02.652539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:02.652551 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:20:02.652588 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:02.652591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:02.652632 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:02.652959 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:02.653066 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:02.653750 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:02.655135 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:02.655790 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:02.655889 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:02.655946 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:02.656328 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 12:20:02.656399 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":1090,"thread_start_us":72,"threads_started":1}
I20260620 12:20:02.656471 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1112e7032e0645afb6074ac15f732b76" is_pre_election: true
I20260620 12:20:02.656756 7290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e06b3f42475f46e4b74cc800eac7b1cf in term 1.
I20260620 12:20:02.656927 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1112e7032e0645afb6074ac15f732b76" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" is_pre_election: true
I20260620 12:20:02.657156 7107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1112e7032e0645afb6074ac15f732b76 in term 1.
I20260620 12:20:02.657181 7082 leader_election.cc:302] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1112e7032e0645afb6074ac15f732b76, e06b3f42475f46e4b74cc800eac7b1cf; no voters: f5e86be99fe3493c8908628297b224cc
I20260620 12:20:02.657449 7804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 12:20:02.657516 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader f5e86be99fe3493c8908628297b224cc)
I20260620 12:20:02.657575 7804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:20:02.657558 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1112e7032e0645afb6074ac15f732b76, e06b3f42475f46e4b74cc800eac7b1cf; no voters: f5e86be99fe3493c8908628297b224cc
I20260620 12:20:02.657763 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 12:20:02.657835 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f5e86be99fe3493c8908628297b224cc)
I20260620 12:20:02.657893 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:20:02.658672 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } }
I20260620 12:20:02.658866 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } }
I20260620 12:20:02.659075 7804 leader_election.cc:288] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [CANDIDATE]: Term 2 election: Requested vote from peers f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687), 1112e7032e0645afb6074ac15f732b76 (127.0.98.60:46505)
I20260620 12:20:02.659231 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 2 election: Requested vote from peers f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687), e06b3f42475f46e4b74cc800eac7b1cf (127.0.98.61:43043)
I20260620 12:20:02.659552 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.659336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44508)
0620 12:20:02.659425 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.659483 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 12:20:02.659741 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.659463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44520)
0620 12:20:02.659562 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.659681 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 12:20:02.659709 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1112e7032e0645afb6074ac15f732b76" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e06b3f42475f46e4b74cc800eac7b1cf"
I20260620 12:20:02.659894 7107 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1112e7032e0645afb6074ac15f732b76 in current term 2: Already voted for candidate e06b3f42475f46e4b74cc800eac7b1cf in this term.
I20260620 12:20:02.659855 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1112e7032e0645afb6074ac15f732b76"
I20260620 12:20:02.659998 7290 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e06b3f42475f46e4b74cc800eac7b1cf in current term 2: Already voted for candidate 1112e7032e0645afb6074ac15f732b76 in this term.
I20260620 12:20:02.660492 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.660179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44536)
0620 12:20:02.660410 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.660450 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260620 12:20:02.660624 7082 leader_election.cc:334] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:02.660681 7082 leader_election.cc:302] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e06b3f42475f46e4b74cc800eac7b1cf; no voters: 1112e7032e0645afb6074ac15f732b76, f5e86be99fe3493c8908628297b224cc
I20260620 12:20:02.660750 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:02.660490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44538)
0620 12:20:02.660613 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:02.660687 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 12:20:02.660853 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 12:20:02.661037 7265 leader_election.cc:334] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:02.661131 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1112e7032e0645afb6074ac15f732b76; no voters: e06b3f42475f46e4b74cc800eac7b1cf, f5e86be99fe3493c8908628297b224cc
I20260620 12:20:02.661260 7803 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 12:20:03.137300 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:03.136776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:46681)
0620 12:20:03.137101 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:03.137180 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
I20260620 12:20:03.137893 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:03.137453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:35125)
0620 12:20:03.137720 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:03.137792 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
I20260620 12:20:03.138695 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:03.138334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:54551)
0620 12:20:03.138576 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:03.138623 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260620 12:20:04.138738 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.138235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:56343)
0620 12:20:04.138537 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.138606 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260620 12:20:04.139299 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.138790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:54123)
0620 12:20:04.139163 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.139225 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260620 12:20:04.139724 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.139335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:37737)
0620 12:20:04.139570 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.139631 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
I20260620 12:20:04.372435 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:04.372570 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } }
I20260620 12:20:04.372931 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687), e06b3f42475f46e4b74cc800eac7b1cf (127.0.98.61:43043)
I20260620 12:20:04.373706 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.373244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44688)
0620 12:20:04.373500 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.373585 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260620 12:20:04.373929 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1112e7032e0645afb6074ac15f732b76" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" is_pre_election: true
I20260620 12:20:04.374087 7107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1112e7032e0645afb6074ac15f732b76 in term 2.
I20260620 12:20:04.374501 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1112e7032e0645afb6074ac15f732b76, e06b3f42475f46e4b74cc800eac7b1cf; no voters:
I20260620 12:20:04.374616 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.374384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44704)
0620 12:20:04.374474 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.374524 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 12:20:04.374668 7819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 12:20:04.374745 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:04.374792 7819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 2 FOLLOWER]: Advancing to term 3
W20260620 12:20:04.374807 7265 leader_election.cc:334] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:04.375795 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } }
I20260620 12:20:04.376111 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 3 election: Requested vote from peers f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687), e06b3f42475f46e4b74cc800eac7b1cf (127.0.98.61:43043)
I20260620 12:20:04.376452 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.376202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44720)
0620 12:20:04.376332 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.376388 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 12:20:04.376667 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1112e7032e0645afb6074ac15f732b76" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e06b3f42475f46e4b74cc800eac7b1cf"
I20260620 12:20:04.376784 7107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:20:04.377178 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.376939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44734)
0620 12:20:04.377041 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.377126 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 12:20:04.377333 7265 leader_election.cc:334] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
I20260620 12:20:04.377940 7107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1112e7032e0645afb6074ac15f732b76 in term 3.
I20260620 12:20:04.378214 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1112e7032e0645afb6074ac15f732b76, e06b3f42475f46e4b74cc800eac7b1cf; no voters: f5e86be99fe3493c8908628297b224cc
I20260620 12:20:04.378358 7819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 12:20:04.378530 7819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [term 3 LEADER]: Becoming Leader. State: Replica: 1112e7032e0645afb6074ac15f732b76, State: Running, Role: LEADER
I20260620 12:20:04.378741 7819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } }
I20260620 12:20:04.380080 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1112e7032e0645afb6074ac15f732b76. Latest consensus state: current_term: 3 leader_uuid: "1112e7032e0645afb6074ac15f732b76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:04.380200 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:04.380499 7824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:20:04.382718 7824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:20:04.383133 7824 catalog_manager.cc:1259] Loaded cluster ID: 24156dbf37c0475cb36282614b8402d0
I20260620 12:20:04.383175 7824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:20:04.383731 7824 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:20:04.384112 7824 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Loaded TSK: 0
I20260620 12:20:04.384768 7824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:20:04.402717 7279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:20:04.405901 7279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:20:04.408893 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.408648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44750)
0620 12:20:04.408783 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.408829 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 12:20:04.408982 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [term 3 FOLLOWER]: Refusing update from remote peer 1112e7032e0645afb6074ac15f732b76: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 12:20:04.409705 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.409487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.0.1:44756)
0620 12:20:04.409597 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.409649 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 12:20:04.409701 7819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 12:20:04.409927 7265 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 -> Peer f5e86be99fe3493c8908628297b224cc (127.0.98.62:37687): Couldn't send request to peer f5e86be99fe3493c8908628297b224cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 12:20:04.411340 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1112e7032e0645afb6074ac15f732b76. Latest consensus state: current_term: 3 leader_uuid: "1112e7032e0645afb6074ac15f732b76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:04.411480 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:20:04.412693 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1112e7032e0645afb6074ac15f732b76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:04.412860 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:04.413062 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1112e7032e0645afb6074ac15f732b76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5e86be99fe3493c8908628297b224cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37687 } } peers { permanent_uuid: "e06b3f42475f46e4b74cc800eac7b1cf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43043 } attrs { promote: false } } peers { permanent_uuid: "1112e7032e0645afb6074ac15f732b76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46505 } attrs { promote: false } } }
I20260620 12:20:04.413203 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06b3f42475f46e4b74cc800eac7b1cf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:20:04.413757 7827 mvcc.cc:205] Tried to move back new op lower bound from 7298899986054086656 to 7298899985937690624. Current Snapshot: MvccSnapshot[applied={T|T < 7298899986054086656 or (T in {7298899986054086656})}]
I20260620 12:20:04.423053 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.418609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43129 (local address 127.0.0.1:54000)
0620 12:20:04.418724 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.418739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:04.418834 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:04.419563 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:04.419567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:04.419594 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:04.419764 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.419770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.420971 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.420974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.421829 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.421837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.422639 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.422642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.422728 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.422753 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:20:04.422823 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 12:20:04.422890 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":33,"spinlock_wait_cycles":6400}
I20260620 12:20:04.423539 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.418738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.0.98.2:43129)
0620 12:20:04.419307 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:04.419311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:04.419327 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:20:04.419382 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:04.419386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:04.419432 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:04.419860 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:04.419984 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.420867 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.421962 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.422506 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.422621 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.422686 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:04.423312 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 12:20:04.423401 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":76,"threads_started":1}
I20260620 12:20:04.423655 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.418939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33747 (local address 127.0.0.1:51942)
0620 12:20:04.419175 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.419190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:04.419269 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:04.419563 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:04.419567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:04.419607 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:04.419810 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.419818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.420858 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.420861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.422145 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.422154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.423031 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.423036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.423130 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.423154 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:20:04.423417 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 12:20:04.423494 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":51,"spinlock_wait_cycles":6656,"thread_start_us":110,"threads_started":1}
I20260620 12:20:04.424064 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.419035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.0.98.3:33747)
0620 12:20:04.419322 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:04.419327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:04.419341 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:04.419384 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:04.419387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:04.419433 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:04.419710 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:04.419928 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.420724 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.422295 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.422930 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.423024 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.423091 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:04.423809 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 12:20:04.423925 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260620 12:20:04.424111 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.419262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.0.1:56552)
0620 12:20:04.419458 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.419471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:04.419547 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:04.420110 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:04.420113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:04.420128 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:04.420214 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.420220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.421148 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.421151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.422269 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.422277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.423647 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.423652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.423837 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.423865 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:20:04.423928 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:20:04.423995 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260620 12:20:04.424340 7530 tablet_service.cc:1467] Processing CreateTablet for tablet e0a4ce1622134675b7c0f1b3f6b16b21 (DEFAULT_TABLE table=test-table [id=10abffbd31c84e0e96e4b8ed423fbda5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:04.424495 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.419613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.0.98.1:35671)
0620 12:20:04.419895 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:04.419899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:04.419913 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:04.419961 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:04.419964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:04.420011 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:04.420102 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:04.420340 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.421033 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.422381 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.423519 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.423617 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.423675 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:04.424289 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 12:20:04.424371 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
I20260620 12:20:04.424628 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a4ce1622134675b7c0f1b3f6b16b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:04.425014 7393 tablet_service.cc:1467] Processing CreateTablet for tablet e0a4ce1622134675b7c0f1b3f6b16b21 (DEFAULT_TABLE table=test-table [id=10abffbd31c84e0e96e4b8ed423fbda5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:04.425331 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a4ce1622134675b7c0f1b3f6b16b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:04.425509 7667 tablet_service.cc:1467] Processing CreateTablet for tablet e0a4ce1622134675b7c0f1b3f6b16b21 (DEFAULT_TABLE table=test-table [id=10abffbd31c84e0e96e4b8ed423fbda5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:04.425766 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a4ce1622134675b7c0f1b3f6b16b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:04.426934 7837 tablet_bootstrap.cc:492] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: Bootstrap starting.
I20260620 12:20:04.427745 7838 tablet_bootstrap.cc:492] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7: Bootstrap starting.
I20260620 12:20:04.427762 7837 tablet_bootstrap.cc:654] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:04.428128 7839 tablet_bootstrap.cc:492] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: Bootstrap starting.
I20260620 12:20:04.428670 7838 tablet_bootstrap.cc:654] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:04.428996 7839 tablet_bootstrap.cc:654] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:04.429853 7838 tablet_bootstrap.cc:492] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7: No bootstrap required, opened a new log
I20260620 12:20:04.429939 7838 ts_tablet_manager.cc:1397] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:04.430151 7839 tablet_bootstrap.cc:492] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: No bootstrap required, opened a new log
I20260620 12:20:04.430249 7839 ts_tablet_manager.cc:1397] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:04.430357 7838 raft_consensus.cc:348] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.430465 7838 raft_consensus.cc:374] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:04.430495 7838 raft_consensus.cc:729] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad5a9dc70444049a8340f57f8928cd7, State: Initialized, Role: FOLLOWER
I20260620 12:20:04.430605 7838 consensus_queue.cc:260] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.430641 7839 raft_consensus.cc:348] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.430733 7839 raft_consensus.cc:374] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:04.430755 7839 raft_consensus.cc:729] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6253039999647e68e450a5d4993cfaf, State: Initialized, Role: FOLLOWER
I20260620 12:20:04.430918 7838 ts_tablet_manager.cc:1428] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:04.430903 7839 consensus_queue.cc:260] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.431263 7839 ts_tablet_manager.cc:1428] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:04.433418 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.431527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:46073)
0620 12:20:04.433299 (+ 1772us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.433356 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1687}
I20260620 12:20:04.433622 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.433377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:38727)
0620 12:20:04.433503 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.433559 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 12:20:04.433813 7459 heartbeater.cc:503] Master 127.0.98.60:46505 was elected leader, sending a full tablet report...
I20260620 12:20:04.434186 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.433939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:50437)
0620 12:20:04.434071 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.434127 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 12:20:04.437597 7837 tablet_bootstrap.cc:492] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: No bootstrap required, opened a new log
I20260620 12:20:04.437721 7837 ts_tablet_manager.cc:1397] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260620 12:20:04.438151 7837 raft_consensus.cc:348] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.438270 7837 raft_consensus.cc:374] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:04.438314 7837 raft_consensus.cc:729] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f788b0584444fcaa8c693eabd2817a, State: Initialized, Role: FOLLOWER
I20260620 12:20:04.438452 7837 consensus_queue.cc:260] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.438791 7837 ts_tablet_manager.cc:1428] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 12:20:04.439154 7733 heartbeater.cc:503] Master 127.0.98.60:46505 was elected leader, sending a full tablet report...
I20260620 12:20:04.439538 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.439318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:46957)
0620 12:20:04.439414 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.439466 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 12:20:04.440338 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.439685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:41925)
0620 12:20:04.440209 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.440266 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456}
I20260620 12:20:04.440341 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.440097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:49289)
0620 12:20:04.440234 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.440283 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 12:20:04.441323 7596 heartbeater.cc:503] Master 127.0.98.60:46505 was elected leader, sending a full tablet report...
I20260620 12:20:04.469827 7845 raft_consensus.cc:482] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:04.469960 7845 raft_consensus.cc:504] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.471038 7845 leader_election.cc:288] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6253039999647e68e450a5d4993cfaf (127.0.98.3:33747), 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671)
I20260620 12:20:04.476480 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.471764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43399 (local address 127.0.98.1:35671)
0620 12:20:04.471879 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:04.471883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:04.471941 (+ 58us) server_negotiation.cc:407] Connection header received
0620 12:20:04.472151 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:04.472154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:04.472225 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:04.472311 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:04.472532 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.473294 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.474967 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.475599 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.475720 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.475783 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:04.476249 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 12:20:04.476334 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:20:04.476532 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.471383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56279 (local address 127.0.98.3:33747)
0620 12:20:04.471524 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:04.471529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:04.471547 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:20:04.471597 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:04.471600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:04.471667 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:04.471747 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:04.472239 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.473127 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.474970 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.475631 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.475725 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.475783 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:04.476249 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 12:20:04.476334 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 12:20:04.476527 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.471246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33747 (local address 127.0.98.2:56279)
0620 12:20:04.471373 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.471389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:04.471500 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:04.471939 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:04.471942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:04.471965 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:04.472111 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.472118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.473234 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.473241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.474833 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.474845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.475808 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.475813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.476015 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.476091 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 12:20:04.476177 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:20:04.476345 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":300}
I20260620 12:20:04.476593 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.471505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.98.2:43399)
0620 12:20:04.471865 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.471878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:04.472001 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:04.472326 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:04.472328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:04.472343 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:04.472424 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.472428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.473394 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.473397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.474833 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:04.474845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.475808 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.475813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:04.476012 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.476088 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 12:20:04.476177 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 12:20:04.476345 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":208,"spinlock_wait_cycles":2432,"thread_start_us":115,"threads_started":1}
I20260620 12:20:04.477174 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a4ce1622134675b7c0f1b3f6b16b21" candidate_uuid: "29f788b0584444fcaa8c693eabd2817a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad5a9dc70444049a8340f57f8928cd7" is_pre_election: true
I20260620 12:20:04.477273 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a4ce1622134675b7c0f1b3f6b16b21" candidate_uuid: "29f788b0584444fcaa8c693eabd2817a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6253039999647e68e450a5d4993cfaf" is_pre_election: true
I20260620 12:20:04.477342 7413 raft_consensus.cc:2454] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f788b0584444fcaa8c693eabd2817a in term 0.
I20260620 12:20:04.477521 7687 raft_consensus.cc:2454] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f788b0584444fcaa8c693eabd2817a in term 0.
I20260620 12:20:04.477725 7485 leader_election.cc:302] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f788b0584444fcaa8c693eabd2817a, 7ad5a9dc70444049a8340f57f8928cd7; no voters:
I20260620 12:20:04.477985 7845 raft_consensus.cc:2781] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:20:04.478071 7845 raft_consensus.cc:482] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:04.478111 7845 raft_consensus.cc:3037] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:04.479125 7845 raft_consensus.cc:504] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.479450 7845 leader_election.cc:288] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [CANDIDATE]: Term 1 election: Requested vote from peers c6253039999647e68e450a5d4993cfaf (127.0.98.3:33747), 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671)
I20260620 12:20:04.479859 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a4ce1622134675b7c0f1b3f6b16b21" candidate_uuid: "29f788b0584444fcaa8c693eabd2817a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6253039999647e68e450a5d4993cfaf"
I20260620 12:20:04.479859 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a4ce1622134675b7c0f1b3f6b16b21" candidate_uuid: "29f788b0584444fcaa8c693eabd2817a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad5a9dc70444049a8340f57f8928cd7"
I20260620 12:20:04.479977 7413 raft_consensus.cc:3037] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:04.479977 7687 raft_consensus.cc:3037] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:04.481097 7413 raft_consensus.cc:2454] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f788b0584444fcaa8c693eabd2817a in term 1.
I20260620 12:20:04.481097 7687 raft_consensus.cc:2454] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f788b0584444fcaa8c693eabd2817a in term 1.
I20260620 12:20:04.481369 7485 leader_election.cc:302] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f788b0584444fcaa8c693eabd2817a, 7ad5a9dc70444049a8340f57f8928cd7; no voters:
I20260620 12:20:04.481534 7845 raft_consensus.cc:2781] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:04.481801 7845 raft_consensus.cc:686] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 1 LEADER]: Becoming Leader. State: Replica: 29f788b0584444fcaa8c693eabd2817a, State: Running, Role: LEADER
I20260620 12:20:04.482240 7845 consensus_queue.cc:237] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260620 12:20:04.482487 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.482262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:33923)
0620 12:20:04.482372 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.482430 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 12:20:04.484809 7280 catalog_manager.cc:5495] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f788b0584444fcaa8c693eabd2817a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "29f788b0584444fcaa8c693eabd2817a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f788b0584444fcaa8c693eabd2817a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } health_report { overall_health: UNKNOWN } } }
I20260620 12:20:04.509464 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.506105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.0.98.2:43129)
0620 12:20:04.506270 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:04.506273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:04.506288 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:20:04.506364 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:04.506366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:04.506426 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:04.506541 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:20:04.506750 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.507322 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.508140 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:04.508357 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:04.508441 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:04.508666 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:20:04.509133 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:20:04.509220 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:04.509309 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 12:20:04.509356 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 12:20:04.512121 7413 raft_consensus.cc:1261] T e0a4ce1622134675b7c0f1b3f6b16b21 P 7ad5a9dc70444049a8340f57f8928cd7 [term 1 FOLLOWER]: Refusing update from remote peer 29f788b0584444fcaa8c693eabd2817a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:04.512430 7687 raft_consensus.cc:1261] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 1 FOLLOWER]: Refusing update from remote peer 29f788b0584444fcaa8c693eabd2817a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:04.512655 7845 consensus_queue.cc:1035] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad5a9dc70444049a8340f57f8928cd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:04.513006 7847 consensus_queue.cc:1035] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6253039999647e68e450a5d4993cfaf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:04.515450 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.515147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:39331)
0620 12:20:04.515312 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.515374 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 12:20:04.516203 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.515852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:39611)
0620 12:20:04.516066 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.516124 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260620 12:20:04.517491 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.517260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:42625)
0620 12:20:04.517369 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.517426 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 12:20:04.518743 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.518444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.1:41147)
0620 12:20:04.518627 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.518681 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 12:20:04.522855 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.522482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.3:56951)
0620 12:20:04.522729 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.522783 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 12:20:04.524763 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.520507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37687 (local address 127.0.98.2:49379)
0620 12:20:04.524638 (+ 4131us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.524694 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4065}
I20260620 12:20:04.525432 7279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:04.525579 7279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:04.529291 7279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1112e7032e0645afb6074ac15f732b76: Sending DeleteTablet for 3 replicas of tablet e0a4ce1622134675b7c0f1b3f6b16b21
I20260620 12:20:04.530119 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a4ce1622134675b7c0f1b3f6b16b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:04 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54000
I20260620 12:20:04.530290 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a4ce1622134675b7c0f1b3f6b16b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:04 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56552
I20260620 12:20:04.530476 7860 tablet_replica.cc:330] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: stopping tablet replica
I20260620 12:20:04.530594 7860 raft_consensus.cc:2229] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:20:04.530615 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a4ce1622134675b7c0f1b3f6b16b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:04 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51942
I20260620 12:20:04.530916 7860 raft_consensus.cc:2258] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:04.530936 7862 tablet_replica.cc:330] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: stopping tablet replica
I20260620 12:20:04.531189 7862 raft_consensus.cc:2229] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:04.531312 7862 raft_consensus.cc:2258] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:04.531409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 7242
I20260620 12:20:04.532229 7860 ts_tablet_manager.cc:1905] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:04.533635 7862 ts_tablet_manager.cc:1905] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:04.534811 7860 ts_tablet_manager.cc:1918] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:04.534904 7860 log.cc:1196] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/e0a4ce1622134675b7c0f1b3f6b16b21
I20260620 12:20:04.535192 7860 ts_tablet_manager.cc:1939] T e0a4ce1622134675b7c0f1b3f6b16b21 P 29f788b0584444fcaa8c693eabd2817a: Deleting consensus metadata
I20260620 12:20:04.535816 7266 catalog_manager.cc:4841] TS 29f788b0584444fcaa8c693eabd2817a (127.0.98.2:43129): tablet e0a4ce1622134675b7c0f1b3f6b16b21 (table test-table [id=10abffbd31c84e0e96e4b8ed423fbda5]) successfully deleted
I20260620 12:20:04.536106 7862 ts_tablet_manager.cc:1918] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:04.536181 7862 log.cc:1196] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/e0a4ce1622134675b7c0f1b3f6b16b21
I20260620 12:20:04.536441 7862 ts_tablet_manager.cc:1939] T e0a4ce1622134675b7c0f1b3f6b16b21 P c6253039999647e68e450a5d4993cfaf: Deleting consensus metadata
I20260620 12:20:04.537081 7266 catalog_manager.cc:4841] TS c6253039999647e68e450a5d4993cfaf (127.0.98.3:33747): tablet e0a4ce1622134675b7c0f1b3f6b16b21 (table test-table [id=10abffbd31c84e0e96e4b8ed423fbda5]) successfully deleted
I20260620 12:20:04.541584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 7467
I20260620 12:20:04.541795 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:04.541448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.0.1:56560)
0620 12:20:04.541572 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:04.541698 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:20:04.543104 7266 catalog_manager.cc:4584] TS 7ad5a9dc70444049a8340f57f8928cd7 (127.0.98.1:35671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0a4ce1622134675b7c0f1b3f6b16b21: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35671: connect: Connection refused (error 111)
I20260620 12:20:04.547719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 7603
I20260620 12:20:04.554899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 6959
I20260620 12:20:04.560416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 7063
2026-06-20T12:20:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6995 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:20:04 dist-test-slave-9b3m krb5kdc[7865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:20:04 dist-test-slave-9b3m krb5kdc[7865](info): set up 2 sockets
Jun 20 12:20:04 dist-test-slave-9b3m krb5kdc[7865](info): commencing operation
krb5kdc: starting...
W20260620 12:20:05.630314 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:20:05 dist-test-slave-9b3m krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:20:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 12:20:05.755904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33035
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:05.863641 7878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:05.863888 7878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:05.863925 7878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:05.867795 7878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:20:05.867848 7878 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 12:20:05.867864 7878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:05.867882 7878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:05.867898 7878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:20:05.867914 7878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:20:05.873137 7878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33035
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:05.874111 7878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:05.875064 7878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:05.883724 7883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:05.883735 7884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:05.883777 7878 server_base.cc:1034] running on GCE node
W20260620 12:20:05.883754 7886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:05.884542 7878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:05.885763 7878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:05.886997 7878 hybrid_clock.cc:648] HybridClock initialized: now 1781958005886965 us; error 49 us; skew 500 ppm
Jun 20 12:20:05 dist-test-slave-9b3m krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:05.889991 7878 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 12:20:05.890893 7878 webserver.cc:460] Webserver started at http://127.0.98.62:45297/ using document root <none> and password file <none>
I20260620 12:20:05.891484 7878 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:05.891577 7878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:05.891808 7878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:05.893954 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "54fbd9ece10546ddb4eb2436f65ef388"
format_stamp: "Formatted at 2026-06-20 12:20:05 on dist-test-slave-9b3m"
server_key: "71e2250e4a02e71e5cf78441b4da7056"
server_key_iv: "96a6b5dce011ed788022bf1e12078ecf"
server_key_version: "encryptionkey@0"
I20260620 12:20:05.894513 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "54fbd9ece10546ddb4eb2436f65ef388"
format_stamp: "Formatted at 2026-06-20 12:20:05 on dist-test-slave-9b3m"
server_key: "71e2250e4a02e71e5cf78441b4da7056"
server_key_iv: "96a6b5dce011ed788022bf1e12078ecf"
server_key_version: "encryptionkey@0"
I20260620 12:20:05.898523 7878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 12:20:05.901295 7893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:05.902567 7878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 12:20:05.902724 7878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "54fbd9ece10546ddb4eb2436f65ef388"
format_stamp: "Formatted at 2026-06-20 12:20:05 on dist-test-slave-9b3m"
server_key: "71e2250e4a02e71e5cf78441b4da7056"
server_key_iv: "96a6b5dce011ed788022bf1e12078ecf"
server_key_version: "encryptionkey@0"
I20260620 12:20:05.902845 7878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:05.916271 7878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:05.919714 7878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:05.919938 7878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:05.928442 7878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33035
I20260620 12:20:05.928470 7955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33035 every 8 connection(s)
I20260620 12:20:05.929446 7878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:20:05.931895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 7878
I20260620 12:20:05.932054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:20:05.932402 392 external_mini_cluster.cc:1428] Setting key 5bc80f246028cd3476ddae6b9ef05a7c
I20260620 12:20:05.932636 7956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:05.939359 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: Bootstrap starting.
Jun 20 12:20:05 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:20:05.942066 7956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:05.943496 7956 log.cc:824] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:05.945991 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: No bootstrap required, opened a new log
I20260620 12:20:05.947319 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:05.933962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.0.98.62:33035)
0620 12:20:05.934411 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:05.934462 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:05.934498 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:20:05.935131 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:05.935148 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:05.935476 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:05.935809 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:05.936625 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:05.937697 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:05.938551 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:05.939069 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:05.939237 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:05.942055 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:05.942060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:05.942072 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:05.942099 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:05.944298 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:05.944803 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:05.944807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:05.944811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:05.944920 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:05.945308 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:05.945311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:05.945312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:05.945741 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:05.945927 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:05.946281 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 12:20:05.946637 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":149,"threads_started":1}
I20260620 12:20:05.949656 7956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fbd9ece10546ddb4eb2436f65ef388" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } }
I20260620 12:20:05.949877 7956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:05.949962 7956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fbd9ece10546ddb4eb2436f65ef388, State: Initialized, Role: FOLLOWER
I20260620 12:20:05.950475 7956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fbd9ece10546ddb4eb2436f65ef388" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } }
I20260620 12:20:05.950604 7956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:20:05.950693 7956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:20:05.950791 7956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:05.951774 7956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fbd9ece10546ddb4eb2436f65ef388" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } }
I20260620 12:20:05.952150 7956 leader_election.cc:302] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54fbd9ece10546ddb4eb2436f65ef388; no voters:
I20260620 12:20:05.952471 7956 leader_election.cc:288] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:20:05.952623 7961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:05.952881 7961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [term 1 LEADER]: Becoming Leader. State: Replica: 54fbd9ece10546ddb4eb2436f65ef388, State: Running, Role: LEADER
I20260620 12:20:05.953420 7961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fbd9ece10546ddb4eb2436f65ef388" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } }
I20260620 12:20:05.953761 7956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:20:05.955343 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54fbd9ece10546ddb4eb2436f65ef388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fbd9ece10546ddb4eb2436f65ef388" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } } }
I20260620 12:20:05.955471 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:05.955807 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54fbd9ece10546ddb4eb2436f65ef388. Latest consensus state: current_term: 1 leader_uuid: "54fbd9ece10546ddb4eb2436f65ef388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fbd9ece10546ddb4eb2436f65ef388" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } } }
I20260620 12:20:05.955902 7972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:20:05.956459 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:05.959486 7972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:20:05.964535 7972 catalog_manager.cc:1347] Generated new cluster ID: 48cb03b2d010476ca2f76ddad23265a9
I20260620 12:20:05.964608 7972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:20:05.976284 7972 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:20:05.977690 7972 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:20:05.998387 7972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: Generated new TSK 0
I20260620 12:20:05.999193 7972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 12:20:06.031991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:06.168253 7982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:06.168586 7982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:06.168668 7982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:06.172957 7982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:06.173048 7982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:06.173134 7982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:20:06.173215 7982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:20:06.178408 7982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:06.179402 7982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:06.180326 7982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:06.189850 7990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:06.189862 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:06.189845 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:06.190254 7982 server_base.cc:1034] running on GCE node
I20260620 12:20:06.190652 7982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:06.191371 7982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:06.192562 7982 hybrid_clock.cc:648] HybridClock initialized: now 1781958006192541 us; error 44 us; skew 500 ppm
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:06.195483 7982 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 12:20:06.196357 7982 webserver.cc:460] Webserver started at http://127.0.98.1:40357/ using document root <none> and password file <none>
I20260620 12:20:06.196920 7982 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:06.196981 7982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:06.197206 7982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:06.199273 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "bf046b837c28404ea4ed4e511ad1afdd"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "a26fe88b7311a5dc84fd9e31e1bc4406"
server_key_iv: "0ea840de369c13c5fd05e3744ab5f6ac"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.199807 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bf046b837c28404ea4ed4e511ad1afdd"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "a26fe88b7311a5dc84fd9e31e1bc4406"
server_key_iv: "0ea840de369c13c5fd05e3744ab5f6ac"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.203397 7982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:20:06.205811 7997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:06.207082 7982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:06.207202 7982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "bf046b837c28404ea4ed4e511ad1afdd"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "a26fe88b7311a5dc84fd9e31e1bc4406"
server_key_iv: "0ea840de369c13c5fd05e3744ab5f6ac"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.207321 7982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:06.221283 7982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:06.224263 7982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:06.224423 7982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:06.225836 7982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:06.225921 7982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:06.225996 7982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:06.226048 7982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:06.236416 7982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39287
I20260620 12:20:06.236450 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39287 every 8 connection(s)
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:20:06.237491 7982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:20:06.239543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 7982
I20260620 12:20:06.239681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:20:06.239991 392 external_mini_cluster.cc:1428] Setting key 8845c2a1593b8ff6aed7b41bcb966e2c
I20260620 12:20:06.241171 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.226591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60605 (local address 127.0.98.62:33035)
0620 12:20:06.226822 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.226827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.227753 (+ 926us) server_negotiation.cc:407] Connection header received
0620 12:20:06.228921 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.228925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.228991 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.229196 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:06.230907 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.231834 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.233653 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.234068 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.234180 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.237243 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:06.237246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:06.237248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:06.237273 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:06.238938 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.239479 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.239483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.239485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.239571 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.239981 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.239985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.239987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.240244 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:06.240354 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.240845 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.240979 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:20:06.242017 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.227126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.1:60605)
0620 12:20:06.227596 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.227643 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.228663 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.229507 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.229515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.230018 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:06.230725 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.230746 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.232013 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.232017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.233531 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.233541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.234228 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.234231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.234426 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.235224 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:06.235246 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:06.237067 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:06.239040 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.239044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.239054 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.239291 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.239669 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.239674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.239676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.239872 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.240350 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:06.240356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:06.240603 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.240785 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.241143 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":130,"threads_started":1}
I20260620 12:20:06.243359 8113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33035
I20260620 12:20:06.243530 8113 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:06.244017 8113 heartbeater.cc:511] Master 127.0.98.62:33035 requested a full tablet report, sending...
I20260620 12:20:06.245649 7909 ts_manager.cc:194] Registered new tserver with Master: bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287)
I20260620 12:20:06.246745 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60605
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 12:20:06.257359 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.249512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58179 (local address 127.0.98.62:33035)
0620 12:20:06.249631 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.249635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.249704 (+ 69us) server_negotiation.cc:407] Connection header received
0620 12:20:06.249836 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.249839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.249878 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.249981 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:06.250434 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.251069 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.251995 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.252227 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.252333 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.253565 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:06.253572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:06.253577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:06.253608 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:06.255358 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.256051 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.256054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.256055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.256130 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.256476 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.256478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.256479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.256643 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:06.256724 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.257128 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.257232 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:20:06.257540 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.249334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.1:58179)
0620 12:20:06.249634 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.249650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.249748 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.250032 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.250035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.250222 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:06.250317 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.250322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.251189 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.251193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.251875 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.251884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.252336 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.252340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.252436 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.252990 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:06.253002 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:06.253429 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:06.255497 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.255503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.255506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.255779 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.256243 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.256248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.256251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.256363 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.256744 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:06.256751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:06.256876 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.257029 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.257248 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 12:20:06.268143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:06.376178 8122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:06.376446 8122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:06.376490 8122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:06.380476 8122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:06.380529 8122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:06.380568 8122 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:20:06.380668 8122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:20:06.385779 8122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:06.386847 8122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:06.387863 8122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:06.397761 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:06.397746 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:06.397838 8122 server_base.cc:1034] running on GCE node
W20260620 12:20:06.397746 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:06.398339 8122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:06.398977 8122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:06.400159 8122 hybrid_clock.cc:648] HybridClock initialized: now 1781958006400124 us; error 47 us; skew 500 ppm
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:06.402938 8122 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 12:20:06.403846 8122 webserver.cc:460] Webserver started at http://127.0.98.2:35041/ using document root <none> and password file <none>
I20260620 12:20:06.404347 8122 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:06.404412 8122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:06.404573 8122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:06.406607 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "6324bb4152364a28b7fa59276986bace"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "c07ac2c150b16beb444784e9e039df13"
server_key_iv: "70502fe9b448bed32c7b0fb9be15eb6d"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.407189 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6324bb4152364a28b7fa59276986bace"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "c07ac2c150b16beb444784e9e039df13"
server_key_iv: "70502fe9b448bed32c7b0fb9be15eb6d"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.410921 8122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:20:06.413388 8137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:06.414558 8122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:20:06.414693 8122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "6324bb4152364a28b7fa59276986bace"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "c07ac2c150b16beb444784e9e039df13"
server_key_iv: "70502fe9b448bed32c7b0fb9be15eb6d"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.414801 8122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:06.427799 8122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:06.431026 8122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:06.431222 8122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:06.432618 8122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:06.432698 8122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:06.432765 8122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:06.432825 8122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:20:06.443742 8122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43101
I20260620 12:20:06.444523 8252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43101 every 8 connection(s)
I20260620 12:20:06.444917 8122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:20:06.446498 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.433377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49039 (local address 127.0.98.62:33035)
0620 12:20:06.433499 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.433503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.434508 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 12:20:06.435436 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.435440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.435496 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.435579 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:06.436875 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.437807 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.439156 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.439457 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.439577 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.442736 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:06.442741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:06.442744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:06.442771 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:06.444318 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.444901 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.444904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.444905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.445027 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.445460 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.445464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.445466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.445730 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:06.445832 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.446260 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.446366 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:20:06.447353 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.433919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.2:49039)
0620 12:20:06.434369 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.434407 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.435229 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.435878 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.435888 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.436227 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:06.436729 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.436748 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.437950 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.437954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.439024 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.439035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.439620 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.439624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.439801 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.440456 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:06.440477 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:06.442558 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:06.444463 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.444468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.444481 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.444797 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.445147 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.445150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.445152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.445333 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.445822 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:06.445826 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:06.446051 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.446203 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.446445 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20260620 12:20:06.448525 8253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33035
I20260620 12:20:06.448681 8253 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:06.449234 8253 heartbeater.cc:511] Master 127.0.98.62:33035 requested a full tablet report, sending...
I20260620 12:20:06.450246 7909 ts_manager.cc:194] Registered new tserver with Master: 6324bb4152364a28b7fa59276986bace (127.0.98.2:43101)
I20260620 12:20:06.451021 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49039
I20260620 12:20:06.454367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 8122
I20260620 12:20:06.454502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:20:06.454844 392 external_mini_cluster.cc:1428] Setting key ea50e8eb7a9b41c16e6daec3ca13f539
I20260620 12:20:06.458027 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.450711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55047 (local address 127.0.98.62:33035)
0620 12:20:06.450832 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.450836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.451008 (+ 172us) server_negotiation.cc:407] Connection header received
0620 12:20:06.451121 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.451124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.451169 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.451251 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:06.451667 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.452163 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.452962 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.453224 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.453330 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.454426 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:06.454429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:06.454432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:06.454456 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:06.456125 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.456598 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.456607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.456609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.456690 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.456966 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.456971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.456973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.457262 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:06.457368 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.457691 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.457816 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:20:06.458601 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.450640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.2:55047)
0620 12:20:06.450910 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.450930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.451048 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.451263 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.451266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.451469 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:06.451559 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.451565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.452281 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.452284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.452843 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.452851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.453332 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.453335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.453434 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.453957 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:06.453970 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:06.454315 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:06.456271 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.456274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.456277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.456500 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.456779 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.456783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.456785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.456868 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.457359 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:06.457362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:06.457478 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.458222 (+ 744us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.458381 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 12:20:06.481804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:06.592540 8262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:06.592789 8262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:06.592864 8262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:06.596938 8262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:06.597021 8262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:06.597133 8262 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 12:20:06.597206 8262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:20:06.602700 8262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:06.603761 8262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:06.604735 8262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:06.614758 8270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:06.614742 8268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:06.614768 8267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:06.615005 8262 server_base.cc:1034] running on GCE node
I20260620 12:20:06.615571 8262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:06.616230 8262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:06.617398 8262 hybrid_clock.cc:648] HybridClock initialized: now 1781958006617382 us; error 31 us; skew 500 ppm
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:06.620357 8262 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 12:20:06.621284 8262 webserver.cc:460] Webserver started at http://127.0.98.3:44351/ using document root <none> and password file <none>
I20260620 12:20:06.621835 8262 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:06.621925 8262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:06.622145 8262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:06.624205 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "5d451f69db264fad9275790a2e8d5296"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "c7b2bb35b89935567bbf3f3dc62979bd"
server_key_iv: "b463c04cc4a775998070f00564905738"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.624927 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5d451f69db264fad9275790a2e8d5296"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "c7b2bb35b89935567bbf3f3dc62979bd"
server_key_iv: "b463c04cc4a775998070f00564905738"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.628722 8262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 12:20:06.631187 8277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:06.632359 8262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:06.632491 8262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "5d451f69db264fad9275790a2e8d5296"
format_stamp: "Formatted at 2026-06-20 12:20:06 on dist-test-slave-9b3m"
server_key: "c7b2bb35b89935567bbf3f3dc62979bd"
server_key_iv: "b463c04cc4a775998070f00564905738"
server_key_version: "encryptionkey@0"
I20260620 12:20:06.632603 8262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:06.651535 8262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:06.654771 8262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:06.655027 8262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:06.656550 8262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:06.656631 8262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:06.656705 8262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:06.656737 8262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:20:06.667635 8262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36901
I20260620 12:20:06.667744 8392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36901 every 8 connection(s)
I20260620 12:20:06.668622 8262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:20:06.670634 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.657748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41359 (local address 127.0.98.62:33035)
0620 12:20:06.657920 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.657924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.658882 (+ 958us) server_negotiation.cc:407] Connection header received
0620 12:20:06.660135 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.660142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.660215 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.660351 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:06.661673 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.662570 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.663866 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.664105 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.664234 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.667061 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:06.667064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:06.667066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:06.667089 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:06.668673 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.669239 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.669243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.669245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.669338 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.669701 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.669706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.669708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.669915 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:06.669993 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.670424 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.670519 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:20:06.671417 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.658236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.3:41359)
0620 12:20:06.658724 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.658765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.659831 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.660592 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.660599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.661058 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:06.661543 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.661556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.662701 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.662706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.663735 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.663744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.664272 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.664276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.664449 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.665026 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:06.665062 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:06.666898 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:06.668791 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.668797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.668810 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.669123 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.669425 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.669428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.669431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.669587 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.670010 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:06.670014 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:06.670217 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.670358 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.670580 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
I20260620 12:20:06.672524 8393 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33035
I20260620 12:20:06.672653 8393 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:06.673197 8393 heartbeater.cc:511] Master 127.0.98.62:33035 requested a full tablet report, sending...
I20260620 12:20:06.674389 7910 ts_manager.cc:194] Registered new tserver with Master: 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:06.675123 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41359
I20260620 12:20:06.678362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 8262
I20260620 12:20:06.678499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:20:06.678869 392 external_mini_cluster.cc:1428] Setting key ed98911f92b31f7c51951517ec035397
I20260620 12:20:06.680954 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:20:06.682113 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.674813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38435 (local address 127.0.98.62:33035)
0620 12:20:06.674941 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.674946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.675149 (+ 203us) server_negotiation.cc:407] Connection header received
0620 12:20:06.675284 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.675287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.675327 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.675419 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:06.675884 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.676422 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.677433 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.677653 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.677739 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.678851 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:06.678854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:06.678856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:06.678876 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:06.680511 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.681026 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.681032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.681036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.681144 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.681423 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.681425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.681426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.681573 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:06.681647 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.681934 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.682015 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:20:06.682318 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.674732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.3:38435)
0620 12:20:06.675068 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.675079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.675188 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.675433 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.675435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.675620 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:06.675744 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.675750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.676555 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.676558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.677315 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.677323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.677745 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.677747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.677808 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.678229 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:06.678247 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:06.678719 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:06.680615 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.680617 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.680618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.680911 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.681233 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:06.681236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:06.681238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:06.681326 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:06.681666 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:06.681671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:06.681773 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.681890 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.682045 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:20:06.706549 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:06.699754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.0.98.62:33035)
0620 12:20:06.699891 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.699895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.699906 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:20:06.699944 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.699946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.699982 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.700048 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:06.700795 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.701378 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.702294 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.702534 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.702619 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.706319 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36156: BlockingRecv error: recv got EOF from 127.0.0.1:36156 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 12:20:06 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:20:06.715591 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.707793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.0.98.62:33035)
0620 12:20:06.707912 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.707915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.708068 (+ 153us) server_negotiation.cc:407] Connection header received
0620 12:20:06.708158 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.708160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.708193 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.708261 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:06.708887 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.709415 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.710176 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.710367 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.710446 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.712726 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:06.712729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:06.712731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:06.712752 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:06.714251 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.714648 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.714650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.714652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.714720 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:06.714957 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:06.714959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:06.714960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:06.715106 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:06.715200 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.715403 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.715486 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:20:06.718433 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:20:06.721500 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:20:06.735029 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.729419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43101 (local address 127.0.0.1:37134)
0620 12:20:06.730549 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.730563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.730642 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.731261 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.731264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.731322 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:06.731429 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.731436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.732692 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.732695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.733720 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.733729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.734490 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.734494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.734602 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.734642 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.734775 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.734888 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":36,"thread_start_us":90,"threads_started":1}
I20260620 12:20:06.735039 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.730330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.98.2:43101)
0620 12:20:06.730703 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.730713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.730759 (+ 46us) server_negotiation.cc:407] Connection header received
0620 12:20:06.730828 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.730833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.731031 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.731169 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:06.731550 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.732568 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.733841 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.734374 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.734491 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.734650 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.734802 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.734903 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":106,"threads_started":1}
I20260620 12:20:06.735556 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.729957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36901 (local address 127.0.0.1:47632)
0620 12:20:06.731031 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.731042 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.731155 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.731565 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.731567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.731582 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:06.731673 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.731679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.732709 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.732713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.733933 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.733940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.735061 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.735065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.735211 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.735234 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.735310 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.735378 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":5,"thread_start_us":72,"threads_started":1}
I20260620 12:20:06.735579 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.730155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.0.98.3:36901)
0620 12:20:06.730883 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.730896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.731127 (+ 231us) server_negotiation.cc:407] Connection header received
0620 12:20:06.731259 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.731264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.731436 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.731566 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:06.731757 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.732584 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.734046 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.734747 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.734855 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.734997 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.735363 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.735450 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":131,"threads_started":1}
I20260620 12:20:06.735555 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.729294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39287 (local address 127.0.0.1:58810)
0620 12:20:06.729955 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.730011 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.730145 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.730813 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.730819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.730843 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:06.730973 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.730979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.732339 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.732342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.733790 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.733800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.735061 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.735065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.735324 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.735342 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.735402 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.735456 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":197,"thread_start_us":67,"threads_started":1}
I20260620 12:20:06.736063 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.729468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.0.98.1:39287)
0620 12:20:06.729700 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.729717 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.730323 (+ 606us) server_negotiation.cc:407] Connection header received
0620 12:20:06.730404 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.730410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.730624 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.730792 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:06.731114 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.732215 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.733928 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.734747 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.734855 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.734997 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.735816 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.735919 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260620 12:20:06.738408 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 7175084c5d654553bdad8c0c218c61f7 (DEFAULT_TABLE table=test-table [id=272fa8342d124850a18084ff78d07d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:06.738405 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 7175084c5d654553bdad8c0c218c61f7 (DEFAULT_TABLE table=test-table [id=272fa8342d124850a18084ff78d07d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:06.738770 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 7175084c5d654553bdad8c0c218c61f7 (DEFAULT_TABLE table=test-table [id=272fa8342d124850a18084ff78d07d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:06.739480 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7175084c5d654553bdad8c0c218c61f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:06.739498 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7175084c5d654553bdad8c0c218c61f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:06.739754 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7175084c5d654553bdad8c0c218c61f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:06.747014 8423 tablet_bootstrap.cc:492] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: Bootstrap starting.
I20260620 12:20:06.748845 8423 tablet_bootstrap.cc:654] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:06.749234 8421 tablet_bootstrap.cc:492] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: Bootstrap starting.
I20260620 12:20:06.749378 8422 tablet_bootstrap.cc:492] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: Bootstrap starting.
I20260620 12:20:06.749786 8423 log.cc:824] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:06.751255 8421 tablet_bootstrap.cc:654] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:06.751619 8422 tablet_bootstrap.cc:654] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:06.751972 8423 tablet_bootstrap.cc:492] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: No bootstrap required, opened a new log
I20260620 12:20:06.752058 8421 log.cc:824] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:06.752149 8423 ts_tablet_manager.cc:1397] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 12:20:06.752410 8422 log.cc:824] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:06.754009 8421 tablet_bootstrap.cc:492] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: No bootstrap required, opened a new log
I20260620 12:20:06.754210 8421 ts_tablet_manager.cc:1397] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:20:06.754522 8422 tablet_bootstrap.cc:492] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: No bootstrap required, opened a new log
I20260620 12:20:06.754724 8422 ts_tablet_manager.cc:1397] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:20:06.756052 8423 raft_consensus.cc:348] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.756321 8423 raft_consensus.cc:374] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:06.756379 8423 raft_consensus.cc:729] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Initialized, Role: FOLLOWER
I20260620 12:20:06.756824 8423 consensus_queue.cc:260] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.757498 8253 heartbeater.cc:503] Master 127.0.98.62:33035 was elected leader, sending a full tablet report...
I20260620 12:20:06.758018 8422 raft_consensus.cc:348] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.758008 8421 raft_consensus.cc:348] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.758227 8422 raft_consensus.cc:374] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:06.758231 8421 raft_consensus.cc:374] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:06.758275 8421 raft_consensus.cc:729] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d451f69db264fad9275790a2e8d5296, State: Initialized, Role: FOLLOWER
I20260620 12:20:06.758293 8422 raft_consensus.cc:729] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf046b837c28404ea4ed4e511ad1afdd, State: Initialized, Role: FOLLOWER
I20260620 12:20:06.758658 8421 consensus_queue.cc:260] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.758708 8422 consensus_queue.cc:260] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.759550 8422 ts_tablet_manager.cc:1428] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:20:06.759569 8421 ts_tablet_manager.cc:1428] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:20:06.759953 8113 heartbeater.cc:503] Master 127.0.98.62:33035 was elected leader, sending a full tablet report...
I20260620 12:20:06.760680 8393 heartbeater.cc:503] Master 127.0.98.62:33035 was elected leader, sending a full tablet report...
I20260620 12:20:06.761394 8423 ts_tablet_manager.cc:1428] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260620 12:20:06.822182 8429 raft_consensus.cc:482] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:06.822455 8429 raft_consensus.cc:504] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.823681 8429 leader_election.cc:288] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6324bb4152364a28b7fa59276986bace (127.0.98.2:43101), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:06.828198 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.824049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46287 (local address 127.0.98.2:43101)
0620 12:20:06.824188 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.824191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.824334 (+ 143us) server_negotiation.cc:407] Connection header received
0620 12:20:06.824675 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.824679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.824738 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.824820 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:06.825299 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.826004 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.827183 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.827589 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.827669 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.827748 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.827976 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.828055 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:20:06.828213 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.824372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36901 (local address 127.0.98.1:58435)
0620 12:20:06.824703 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.824717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.824822 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.825135 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.825138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.825157 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:06.825279 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.825287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.826203 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.826206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.827042 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.827050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.827783 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.827787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.827887 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.827903 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.828001 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.828062 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":24,"thread_start_us":124,"threads_started":1}
I20260620 12:20:06.828332 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.824548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58435 (local address 127.0.98.3:36901)
0620 12:20:06.824703 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.824707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.824814 (+ 107us) server_negotiation.cc:407] Connection header received
0620 12:20:06.824976 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.824979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.825029 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.825123 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:06.825386 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.826101 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.827174 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.827624 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.827763 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.827828 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.828043 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.828211 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:20:06.828346 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.823931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43101 (local address 127.0.98.1:46287)
0620 12:20:06.824215 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:06.824234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:06.824358 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:06.824891 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:06.824894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:06.824917 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:06.825123 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.825131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.826112 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.826116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.827054 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:06.827064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.827719 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.827727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:06.827831 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.827865 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:20:06.827945 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 12:20:06.828147 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":2,"thread_start_us":117,"threads_started":1}
I20260620 12:20:06.829082 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7175084c5d654553bdad8c0c218c61f7" candidate_uuid: "bf046b837c28404ea4ed4e511ad1afdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296" is_pre_election: true
I20260620 12:20:06.829260 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7175084c5d654553bdad8c0c218c61f7" candidate_uuid: "bf046b837c28404ea4ed4e511ad1afdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6324bb4152364a28b7fa59276986bace" is_pre_election: true
I20260620 12:20:06.829397 8347 raft_consensus.cc:2454] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf046b837c28404ea4ed4e511ad1afdd in term 0.
I20260620 12:20:06.829596 8207 raft_consensus.cc:2454] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf046b837c28404ea4ed4e511ad1afdd in term 0.
I20260620 12:20:06.829948 7998 leader_election.cc:302] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d451f69db264fad9275790a2e8d5296, bf046b837c28404ea4ed4e511ad1afdd; no voters:
I20260620 12:20:06.830207 8429 raft_consensus.cc:2781] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:20:06.830292 8429 raft_consensus.cc:482] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:06.830336 8429 raft_consensus.cc:3037] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:06.831386 8429 raft_consensus.cc:504] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.831714 8429 leader_election.cc:288] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [CANDIDATE]: Term 1 election: Requested vote from peers 6324bb4152364a28b7fa59276986bace (127.0.98.2:43101), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:06.832029 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7175084c5d654553bdad8c0c218c61f7" candidate_uuid: "bf046b837c28404ea4ed4e511ad1afdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6324bb4152364a28b7fa59276986bace"
I20260620 12:20:06.832074 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7175084c5d654553bdad8c0c218c61f7" candidate_uuid: "bf046b837c28404ea4ed4e511ad1afdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296"
I20260620 12:20:06.832166 8207 raft_consensus.cc:3037] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:06.832190 8347 raft_consensus.cc:3037] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:06.833298 8347 raft_consensus.cc:2454] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf046b837c28404ea4ed4e511ad1afdd in term 1.
I20260620 12:20:06.833595 7998 leader_election.cc:302] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d451f69db264fad9275790a2e8d5296, bf046b837c28404ea4ed4e511ad1afdd; no voters:
I20260620 12:20:06.833765 8429 raft_consensus.cc:2781] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:06.833843 8207 raft_consensus.cc:2454] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf046b837c28404ea4ed4e511ad1afdd in term 1.
I20260620 12:20:06.833984 8429 raft_consensus.cc:686] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 1 LEADER]: Becoming Leader. State: Replica: bf046b837c28404ea4ed4e511ad1afdd, State: Running, Role: LEADER
I20260620 12:20:06.834298 8429 consensus_queue.cc:237] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.837594 7909 catalog_manager.cc:5495] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd reported cstate change: term changed from 0 to 1, leader changed from <none> to bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bf046b837c28404ea4ed4e511ad1afdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20260620 12:20:06.857601 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:06.853909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.0.98.1:39287)
0620 12:20:06.854053 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:06.854056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:06.854072 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:20:06.854191 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:06.854194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:06.854247 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:06.854371 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:20:06.854619 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.855179 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.856114 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:06.856355 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:06.856434 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:06.856680 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:20:06.857159 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:20:06.857243 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:06.857367 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 12:20:06.857438 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:20:06.863368 8207 raft_consensus.cc:1261] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Refusing update from remote peer bf046b837c28404ea4ed4e511ad1afdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:06.863576 8347 raft_consensus.cc:1261] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Refusing update from remote peer bf046b837c28404ea4ed4e511ad1afdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:06.864084 8432 consensus_queue.cc:1035] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:06.864264 8429 consensus_queue.cc:1035] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:06.873907 8436 mvcc.cc:205] Tried to move back new op lower bound from 7298899996104941568 to 7298899995996004352. Current Snapshot: MvccSnapshot[applied={T|T < 7298899996104941568 or (T in {7298899996104941568})}]
I20260620 12:20:06.874119 8437 mvcc.cc:205] Tried to move back new op lower bound from 7298899996104941568 to 7298899995996004352. Current Snapshot: MvccSnapshot[applied={T|T < 7298899996104941568}]
I20260620 12:20:06.880544 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:06.880708 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:06.883154 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: Sending DeleteTablet for 3 replicas of tablet 7175084c5d654553bdad8c0c218c61f7
I20260620 12:20:06.883924 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 7175084c5d654553bdad8c0c218c61f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58810
I20260620 12:20:06.884093 8187 tablet_service.cc:1514] Processing DeleteTablet for tablet 7175084c5d654553bdad8c0c218c61f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37134
I20260620 12:20:06.884227 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 7175084c5d654553bdad8c0c218c61f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47632
I20260620 12:20:06.884750 8446 tablet_replica.cc:330] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: stopping tablet replica
I20260620 12:20:06.884771 8445 tablet_replica.cc:330] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: stopping tablet replica
I20260620 12:20:06.885097 8445 raft_consensus.cc:2229] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:20:06.885131 8446 raft_consensus.cc:2229] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:06.885504 8445 raft_consensus.cc:2258] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:06.885427 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:20:06.885780 8446 raft_consensus.cc:2258] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 12:20:06.885879 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:20:06.886776 8447 tablet_replica.cc:330] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: stopping tablet replica
I20260620 12:20:06.887091 8447 raft_consensus.cc:2229] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:06.887405 8447 raft_consensus.cc:2258] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:06.887707 8446 ts_tablet_manager.cc:1905] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:06.888814 8447 ts_tablet_manager.cc:1905] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:06.890761 8446 ts_tablet_manager.cc:1918] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:06.890951 8446 log.cc:1196] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/7175084c5d654553bdad8c0c218c61f7
I20260620 12:20:06.891360 8446 ts_tablet_manager.cc:1939] T 7175084c5d654553bdad8c0c218c61f7 P 6324bb4152364a28b7fa59276986bace: Deleting consensus metadata
I20260620 12:20:06.891808 8447 ts_tablet_manager.cc:1918] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:06.891912 8447 log.cc:1196] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/7175084c5d654553bdad8c0c218c61f7
I20260620 12:20:06.892040 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 293da0a8b16c41058465dd5d1c64d033 (DEFAULT_TABLE table=test-table [id=e5da100ac22d4f08bf0305c0806c3965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:06.892252 8447 ts_tablet_manager.cc:1939] T 7175084c5d654553bdad8c0c218c61f7 P 5d451f69db264fad9275790a2e8d5296: Deleting consensus metadata
I20260620 12:20:06.892306 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293da0a8b16c41058465dd5d1c64d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:06.893077 7894 catalog_manager.cc:4841] TS 6324bb4152364a28b7fa59276986bace (127.0.98.2:43101): tablet 7175084c5d654553bdad8c0c218c61f7 (table test-table [id=272fa8342d124850a18084ff78d07d66]) successfully deleted
I20260620 12:20:06.894465 8423 tablet_bootstrap.cc:492] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: Bootstrap starting.
I20260620 12:20:06.894570 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 293da0a8b16c41058465dd5d1c64d033 (DEFAULT_TABLE table=test-table [id=e5da100ac22d4f08bf0305c0806c3965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:06.894790 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293da0a8b16c41058465dd5d1c64d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:06.895251 8423 tablet_bootstrap.cc:654] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:06.896481 8423 tablet_bootstrap.cc:492] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: No bootstrap required, opened a new log
I20260620 12:20:06.896602 8423 ts_tablet_manager.cc:1397] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 12:20:06.896869 8421 tablet_bootstrap.cc:492] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: Bootstrap starting.
I20260620 12:20:06.897133 8423 raft_consensus.cc:348] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.897259 8423 raft_consensus.cc:374] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:06.897300 8423 raft_consensus.cc:729] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Initialized, Role: FOLLOWER
I20260620 12:20:06.897433 8423 consensus_queue.cc:260] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.897800 8421 tablet_bootstrap.cc:654] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:06.897799 8423 ts_tablet_manager.cc:1428] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:06.898471 7894 catalog_manager.cc:4841] TS 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901): tablet 7175084c5d654553bdad8c0c218c61f7 (table test-table [id=272fa8342d124850a18084ff78d07d66]) successfully deleted
I20260620 12:20:06.898959 8421 tablet_bootstrap.cc:492] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: No bootstrap required, opened a new log
I20260620 12:20:06.899045 8421 ts_tablet_manager.cc:1397] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:06.899322 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 293da0a8b16c41058465dd5d1c64d033 (DEFAULT_TABLE table=test-table [id=e5da100ac22d4f08bf0305c0806c3965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:06.899430 8421 raft_consensus.cc:348] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.899509 8421 raft_consensus.cc:374] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:06.899530 8421 raft_consensus.cc:729] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d451f69db264fad9275790a2e8d5296, State: Initialized, Role: FOLLOWER
I20260620 12:20:06.899578 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293da0a8b16c41058465dd5d1c64d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:06.899655 8421 consensus_queue.cc:260] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.899911 8421 ts_tablet_manager.cc:1428] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:20:06.902101 8422 tablet_bootstrap.cc:492] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: Bootstrap starting.
I20260620 12:20:06.903026 8422 tablet_bootstrap.cc:654] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:06.904340 8422 tablet_bootstrap.cc:492] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: No bootstrap required, opened a new log
I20260620 12:20:06.904434 8422 ts_tablet_manager.cc:1397] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:06.904700 8445 ts_tablet_manager.cc:1905] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:06.904953 8422 raft_consensus.cc:348] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.905109 8422 raft_consensus.cc:374] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:06.905184 8422 raft_consensus.cc:729] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf046b837c28404ea4ed4e511ad1afdd, State: Initialized, Role: FOLLOWER
I20260620 12:20:06.905328 8422 consensus_queue.cc:260] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:06.905646 8422 ts_tablet_manager.cc:1428] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:06.908766 8445 ts_tablet_manager.cc:1918] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:06.908885 8445 log.cc:1196] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/7175084c5d654553bdad8c0c218c61f7
I20260620 12:20:06.909205 8445 ts_tablet_manager.cc:1939] T 7175084c5d654553bdad8c0c218c61f7 P bf046b837c28404ea4ed4e511ad1afdd: Deleting consensus metadata
I20260620 12:20:06.910319 7895 catalog_manager.cc:4841] TS bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287): tablet 7175084c5d654553bdad8c0c218c61f7 (table test-table [id=272fa8342d124850a18084ff78d07d66]) successfully deleted
W20260620 12:20:06.919598 8394 tablet.cc:2369] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:20:06.946100 8254 tablet.cc:2369] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:20:06.989213 8114 tablet.cc:2369] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:20:07.123510 8427 raft_consensus.cc:482] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:07.123678 8427 raft_consensus.cc:504] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.124765 8427 leader_election.cc:288] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:07.128726 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.125295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54685 (local address 127.0.98.1:39287)
0620 12:20:07.125496 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.125501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.125517 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:20:07.125671 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.125674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.125724 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.125809 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:07.126072 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.126576 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.127659 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.128059 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.128177 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.128280 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:07.128513 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 12:20:07.128583 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 12:20:07.128814 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.125113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36901 (local address 127.0.98.2:42997)
0620 12:20:07.125461 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:07.125476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:07.125620 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:07.125883 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:07.125886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:07.125902 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:07.126016 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.126021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.126736 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.126739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.127541 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.127550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.128182 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.128186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.128406 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.128423 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:20:07.128514 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:20:07.128591 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":137,"thread_start_us":98,"threads_started":1}
I20260620 12:20:07.128881 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.125215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42997 (local address 127.0.98.3:36901)
0620 12:20:07.125454 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.125459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.125567 (+ 108us) server_negotiation.cc:407] Connection header received
0620 12:20:07.125721 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.125724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.125781 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.125871 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:07.126120 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.126610 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.127668 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.128055 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.128177 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.128260 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:07.128530 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 12:20:07.128758 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 12:20:07.129416 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.125113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39287 (local address 127.0.98.2:54685)
0620 12:20:07.125404 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:07.125421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:07.125544 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:07.125821 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:07.125824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:07.125839 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:07.125963 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.125968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.126688 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.126694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.127534 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.127543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.128182 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.128186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.128385 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.128405 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:20:07.128495 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:20:07.128570 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":51,"thread_start_us":73,"threads_started":1}
I20260620 12:20:07.129451 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293da0a8b16c41058465dd5d1c64d033" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296" is_pre_election: true
I20260620 12:20:07.129643 8347 raft_consensus.cc:2454] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 0.
I20260620 12:20:07.129938 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293da0a8b16c41058465dd5d1c64d033" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf046b837c28404ea4ed4e511ad1afdd" is_pre_election: true
I20260620 12:20:07.130002 8138 leader_election.cc:302] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d451f69db264fad9275790a2e8d5296, 6324bb4152364a28b7fa59276986bace; no voters:
I20260620 12:20:07.130173 8067 raft_consensus.cc:2454] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 0.
I20260620 12:20:07.130287 8427 raft_consensus.cc:2781] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:20:07.130349 8427 raft_consensus.cc:482] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:07.130402 8427 raft_consensus.cc:3037] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.131457 8427 raft_consensus.cc:504] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.131799 8427 leader_election.cc:288] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 election: Requested vote from peers bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:07.132160 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293da0a8b16c41058465dd5d1c64d033" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296"
I20260620 12:20:07.132310 8347 raft_consensus.cc:3037] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.133184 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293da0a8b16c41058465dd5d1c64d033" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf046b837c28404ea4ed4e511ad1afdd"
I20260620 12:20:07.133335 8067 raft_consensus.cc:3037] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.133363 8347 raft_consensus.cc:2454] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 1.
I20260620 12:20:07.133690 8138 leader_election.cc:302] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d451f69db264fad9275790a2e8d5296, 6324bb4152364a28b7fa59276986bace; no voters:
I20260620 12:20:07.133870 8427 raft_consensus.cc:2781] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:07.134038 8427 raft_consensus.cc:686] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 1 LEADER]: Becoming Leader. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Running, Role: LEADER
I20260620 12:20:07.134289 8427 consensus_queue.cc:237] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.134624 8067 raft_consensus.cc:2454] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 1.
I20260620 12:20:07.136781 7908 catalog_manager.cc:5495] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace reported cstate change: term changed from 0 to 1, leader changed from <none> to 6324bb4152364a28b7fa59276986bace (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } health_report { overall_health: UNKNOWN } } }
Jun 20 12:20:07 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:20:07.231045 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.222800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.0.1:36160)
0620 12:20:07.223100 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:07.223114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:07.223212 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:07.223471 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:07.223473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:07.223870 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:07.223959 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.223965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.224692 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.224695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.225310 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.225318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.225733 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.225735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.225794 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.226301 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:07.226312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:07.227697 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:07.229665 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:07.229673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:07.229678 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:07.229896 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:07.230133 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:07.230135 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:07.230136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:07.230209 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:07.230599 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:07.230605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:07.230701 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 12:20:07.230793 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 12:20:07.230912 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260620 12:20:07.231129 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.222875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.0.98.62:33035)
0620 12:20:07.223174 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.223181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.223196 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:20:07.223312 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.223314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.223353 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.223431 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:07.224068 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.224605 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.225412 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.225632 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.225707 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.227829 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:07.227837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:07.227839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:07.227869 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:07.229566 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:07.229979 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:07.229981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:07.229983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:07.230058 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:07.230316 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:07.230320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:07.230321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:07.230497 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:07.230580 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:07.230875 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 12:20:07.230985 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":135,"threads_started":1}
I20260620 12:20:07.234925 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36160:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 12:20:07.235674 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:20:07.240727 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3a632b5d11426687fd272d8d118418 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e0f67043f9fd41d982aae9aee945f159]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:20:07.241091 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3a632b5d11426687fd272d8d118418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.240919 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3a632b5d11426687fd272d8d118418 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e0f67043f9fd41d982aae9aee945f159]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:20:07.240991 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3a632b5d11426687fd272d8d118418 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e0f67043f9fd41d982aae9aee945f159]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 12:20:07.241324 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3a632b5d11426687fd272d8d118418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.241503 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3a632b5d11426687fd272d8d118418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.243750 8422 tablet_bootstrap.cc:492] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: Bootstrap starting.
I20260620 12:20:07.244213 8423 tablet_bootstrap.cc:492] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: Bootstrap starting.
I20260620 12:20:07.244540 8422 tablet_bootstrap.cc:654] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.245146 8423 tablet_bootstrap.cc:654] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.245616 8422 tablet_bootstrap.cc:492] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: No bootstrap required, opened a new log
I20260620 12:20:07.245697 8422 ts_tablet_manager.cc:1397] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:07.246039 8421 tablet_bootstrap.cc:492] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: Bootstrap starting.
I20260620 12:20:07.246044 8422 raft_consensus.cc:348] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.246148 8422 raft_consensus.cc:374] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.246178 8422 raft_consensus.cc:729] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf046b837c28404ea4ed4e511ad1afdd, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.246291 8422 consensus_queue.cc:260] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.246538 8422 ts_tablet_manager.cc:1428] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:20:07.246946 8421 tablet_bootstrap.cc:654] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.247534 8440 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.247699 8440 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:20:07.248605 8421 tablet_bootstrap.cc:492] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: No bootstrap required, opened a new log
I20260620 12:20:07.248673 8421 ts_tablet_manager.cc:1397] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 12:20:07.249117 8421 raft_consensus.cc:348] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.249215 8421 raft_consensus.cc:374] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.249238 8421 raft_consensus.cc:729] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d451f69db264fad9275790a2e8d5296, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.249364 8421 consensus_queue.cc:260] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.249538 8423 tablet_bootstrap.cc:492] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: No bootstrap required, opened a new log
I20260620 12:20:07.249604 8423 ts_tablet_manager.cc:1397] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260620 12:20:07.249644 8421 ts_tablet_manager.cc:1428] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:07.249585 8428 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.249711 8428 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:20:07.249979 8423 raft_consensus.cc:348] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.250069 8423 raft_consensus.cc:374] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.250097 8423 raft_consensus.cc:729] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.250205 8423 consensus_queue.cc:260] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.250433 8453 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.250581 8453 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:20:07.250422 8423 ts_tablet_manager.cc:1428] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:20:07.462167 8453 raft_consensus.cc:482] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:07.462288 8453 raft_consensus.cc:504] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.462759 8453 leader_election.cc:288] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:07.463220 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3a632b5d11426687fd272d8d118418" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf046b837c28404ea4ed4e511ad1afdd" is_pre_election: true
I20260620 12:20:07.463253 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3a632b5d11426687fd272d8d118418" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296" is_pre_election: true
I20260620 12:20:07.463387 8067 raft_consensus.cc:2454] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 0.
I20260620 12:20:07.463441 8347 raft_consensus.cc:2454] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 0.
I20260620 12:20:07.463723 8139 leader_election.cc:302] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6324bb4152364a28b7fa59276986bace, bf046b837c28404ea4ed4e511ad1afdd; no voters:
I20260620 12:20:07.463923 8453 raft_consensus.cc:2781] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:20:07.464030 8453 raft_consensus.cc:482] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:07.464088 8453 raft_consensus.cc:3037] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.465109 8453 raft_consensus.cc:504] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.465487 8453 leader_election.cc:288] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 election: Requested vote from peers bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:07.465868 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3a632b5d11426687fd272d8d118418" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296"
I20260620 12:20:07.465849 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3a632b5d11426687fd272d8d118418" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf046b837c28404ea4ed4e511ad1afdd"
I20260620 12:20:07.465989 8347 raft_consensus.cc:3037] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.465989 8067 raft_consensus.cc:3037] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.467110 8067 raft_consensus.cc:2454] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 1.
I20260620 12:20:07.467109 8347 raft_consensus.cc:2454] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 1.
I20260620 12:20:07.467398 8139 leader_election.cc:302] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6324bb4152364a28b7fa59276986bace, bf046b837c28404ea4ed4e511ad1afdd; no voters:
I20260620 12:20:07.467586 8453 raft_consensus.cc:2781] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:07.467667 8453 raft_consensus.cc:686] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [term 1 LEADER]: Becoming Leader. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Running, Role: LEADER
I20260620 12:20:07.467830 8453 consensus_queue.cc:237] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.468497 8427 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: TxnStatusTablet state changed. Reason: New leader 6324bb4152364a28b7fa59276986bace. Latest consensus state: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.468624 8427 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:20:07.468842 8466 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 12:20:07.469008 8466 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 12:20:07.469705 7908 catalog_manager.cc:5495] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace reported cstate change: term changed from 0 to 1, leader changed from <none> to 6324bb4152364a28b7fa59276986bace (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20260620 12:20:07.492164 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.488354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43101 (local address 127.0.0.1:35450)
0620 12:20:07.488502 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:07.488519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:07.488646 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:07.489000 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:07.489003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:07.489010 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:20:07.489156 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.489162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.489946 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.489949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.490558 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.490566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.491060 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.491064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.491164 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.491205 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:20:07.491867 (+ 662us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:20:07.491881 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 12:20:07.491950 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 12:20:07.492039 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 12:20:07.492200 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.488464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.0.98.2:43101)
0620 12:20:07.488710 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.488725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.488740 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:20:07.488812 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.488814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.488865 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.488991 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:20:07.489292 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.489832 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.490677 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.490946 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.491059 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.491354 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:20:07.491766 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:20:07.491891 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:07.491959 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 12:20:07.492012 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260620 12:20:07.495707 8347 raft_consensus.cc:1261] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Refusing update from remote peer 6324bb4152364a28b7fa59276986bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:07.495887 8067 raft_consensus.cc:1261] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Refusing update from remote peer 6324bb4152364a28b7fa59276986bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:07.496268 8427 consensus_queue.cc:1035] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:07.496490 8453 consensus_queue.cc:1035] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:07.497941 8428 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: TxnStatusTablet state changed. Reason: New leader 6324bb4152364a28b7fa59276986bace. Latest consensus state: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.498070 8428 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:20:07.498220 8440 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: TxnStatusTablet state changed. Reason: New leader 6324bb4152364a28b7fa59276986bace. Latest consensus state: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.498324 8440 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:20:07.501168 8427 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.501282 8427 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:20:07.501727 8440 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.501837 8440 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P bf046b837c28404ea4ed4e511ad1afdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:20:07.502398 8453 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.502491 8453 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P 6324bb4152364a28b7fa59276986bace: This TxnStatusTablet replica's current role is: LEADER
I20260620 12:20:07.503568 8428 tablet_replica.cc:439] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } } }
I20260620 12:20:07.503676 8428 tablet_replica.cc:442] T 4e3a632b5d11426687fd272d8d118418 P 5d451f69db264fad9275790a2e8d5296: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 12:20:07.504873 8472 mvcc.cc:205] Tried to move back new op lower bound from 7298899998696820736 to 7298899998589624320. Current Snapshot: MvccSnapshot[applied={T|T < 7298899998696820736 or (T in {7298899998696820736})}]
I20260620 12:20:07.512919 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.509196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.0.98.2:43101)
0620 12:20:07.509322 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.509326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.509571 (+ 245us) server_negotiation.cc:407] Connection header received
0620 12:20:07.509680 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.509682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.509721 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.509778 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:20:07.510010 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.510735 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.511929 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.512135 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.512215 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.512499 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:20:07.512564 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:20:07.512621 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:07.512738 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 12:20:07.512778 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:20:07.523242 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.519968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33377 (local address 127.0.98.2:43101)
0620 12:20:07.520101 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.520107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.520291 (+ 184us) server_negotiation.cc:407] Connection header received
0620 12:20:07.520441 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.520445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.520489 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.520566 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:20:07.520796 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.521291 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.522149 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.522363 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.522467 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.522764 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:20:07.522838 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:20:07.522917 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:07.523016 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 12:20:07.523088 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":36}
I20260620 12:20:07.523249 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.519931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43101 (local address 127.0.98.2:33377)
0620 12:20:07.520216 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:07.520228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:07.520345 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:07.520574 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:07.520576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:07.520583 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 12:20:07.520691 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.520697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.521399 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.521404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.522039 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:07.522047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.522477 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.522480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:07.522600 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.522642 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 12:20:07.522920 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 12:20:07.522927 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:20:07.523008 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:20:07.523111 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":27,"thread_start_us":92,"threads_started":1}
I20260620 12:20:07.530174 8347 raft_consensus.cc:1261] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Refusing update from remote peer 6324bb4152364a28b7fa59276986bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:07.530296 8067 raft_consensus.cc:1261] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Refusing update from remote peer 6324bb4152364a28b7fa59276986bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:07.530606 8484 consensus_queue.cc:1035] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:07.530815 8427 consensus_queue.cc:1035] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:07.567874 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:07.568055 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:07.570258 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: Sending DeleteTablet for 3 replicas of tablet 293da0a8b16c41058465dd5d1c64d033
I20260620 12:20:07.570812 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 293da0a8b16c41058465dd5d1c64d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58810
I20260620 12:20:07.570936 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 293da0a8b16c41058465dd5d1c64d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37134
I20260620 12:20:07.571162 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 293da0a8b16c41058465dd5d1c64d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47632
I20260620 12:20:07.571332 8489 tablet_replica.cc:330] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: stopping tablet replica
I20260620 12:20:07.571499 8489 raft_consensus.cc:2229] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:20:07.571624 8490 tablet_replica.cc:330] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: stopping tablet replica
I20260620 12:20:07.571805 8489 raft_consensus.cc:2258] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:07.571815 8490 raft_consensus.cc:2229] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:07.572151 8488 tablet_replica.cc:330] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: stopping tablet replica
I20260620 12:20:07.572857 8490 raft_consensus.cc:2258] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:07.574067 8489 ts_tablet_manager.cc:1905] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:07.575318 8488 raft_consensus.cc:2229] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:07.575455 8488 raft_consensus.cc:2258] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:07.576108 8488 ts_tablet_manager.cc:1905] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:07.576864 8489 ts_tablet_manager.cc:1918] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:20:07.576970 8489 log.cc:1196] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/293da0a8b16c41058465dd5d1c64d033
I20260620 12:20:07.577261 8489 ts_tablet_manager.cc:1939] T 293da0a8b16c41058465dd5d1c64d033 P 6324bb4152364a28b7fa59276986bace: Deleting consensus metadata
I20260620 12:20:07.573745 8490 ts_tablet_manager.cc:1905] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:07.578011 7894 catalog_manager.cc:4841] TS 6324bb4152364a28b7fa59276986bace (127.0.98.2:43101): tablet 293da0a8b16c41058465dd5d1c64d033 (table test-table [id=e5da100ac22d4f08bf0305c0806c3965]) successfully deleted
I20260620 12:20:07.578913 8488 ts_tablet_manager.cc:1918] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:20:07.578994 8488 log.cc:1196] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/293da0a8b16c41058465dd5d1c64d033
I20260620 12:20:07.579226 8488 ts_tablet_manager.cc:1939] T 293da0a8b16c41058465dd5d1c64d033 P bf046b837c28404ea4ed4e511ad1afdd: Deleting consensus metadata
I20260620 12:20:07.579954 7895 catalog_manager.cc:4841] TS bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287): tablet 293da0a8b16c41058465dd5d1c64d033 (table test-table [id=e5da100ac22d4f08bf0305c0806c3965]) successfully deleted
I20260620 12:20:07.580431 8490 ts_tablet_manager.cc:1918] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:20:07.580534 8490 log.cc:1196] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/293da0a8b16c41058465dd5d1c64d033
I20260620 12:20:07.580816 8490 ts_tablet_manager.cc:1939] T 293da0a8b16c41058465dd5d1c64d033 P 5d451f69db264fad9275790a2e8d5296: Deleting consensus metadata
I20260620 12:20:07.581502 7894 catalog_manager.cc:4841] TS 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901): tablet 293da0a8b16c41058465dd5d1c64d033 (table test-table [id=e5da100ac22d4f08bf0305c0806c3965]) successfully deleted
Jun 20 12:20:07 dist-test-slave-9b3m krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 12:20:07 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:07 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:20:07.603684 8461 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:07.597318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.0.98.62:33035)
0620 12:20:07.597448 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.597454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.597634 (+ 180us) server_negotiation.cc:407] Connection header received
0620 12:20:07.597781 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.597784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.597841 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.597929 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:07.598646 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.599210 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.600068 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.600305 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.600417 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.603546 (+ 3129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58154: BlockingRecv error: recv got EOF from 127.0.0.1:58154 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 12:20:07 dist-test-slave-9b3m krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 12:20:07.614497 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.605393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.0.98.62:33035)
0620 12:20:07.605501 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.605507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.605583 (+ 76us) server_negotiation.cc:407] Connection header received
0620 12:20:07.605710 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.605713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.605759 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.605851 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:07.606523 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.607061 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.607861 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.608076 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.608181 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.610916 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:07.610919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:07.610922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:07.610979 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:07.612792 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:07.613274 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:07.613280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:07.613282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:07.613362 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:07.613637 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:07.613642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:07.613644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:07.613822 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:07.613961 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:07.614202 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 12:20:07.614328 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:20:07.617208 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:20:07.617700 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:20:07.625303 8047 tablet_service.cc:1467] Processing CreateTablet for tablet f67718db09924e0388e11845d1ed95da (DEFAULT_TABLE table=test-table [id=016a960e173345fd8e63c419ecb866e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:07.625718 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67718db09924e0388e11845d1ed95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.628113 8422 tablet_bootstrap.cc:492] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: Bootstrap starting.
I20260620 12:20:07.628628 8186 tablet_service.cc:1467] Processing CreateTablet for tablet f67718db09924e0388e11845d1ed95da (DEFAULT_TABLE table=test-table [id=016a960e173345fd8e63c419ecb866e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:07.628857 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67718db09924e0388e11845d1ed95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.629336 8327 tablet_service.cc:1467] Processing CreateTablet for tablet f67718db09924e0388e11845d1ed95da (DEFAULT_TABLE table=test-table [id=016a960e173345fd8e63c419ecb866e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:07.629604 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67718db09924e0388e11845d1ed95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.631480 8423 tablet_bootstrap.cc:492] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: Bootstrap starting.
I20260620 12:20:07.632480 8423 tablet_bootstrap.cc:654] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.632915 8421 tablet_bootstrap.cc:492] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: Bootstrap starting.
I20260620 12:20:07.633930 8423 tablet_bootstrap.cc:492] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: No bootstrap required, opened a new log
I20260620 12:20:07.633967 8421 tablet_bootstrap.cc:654] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.633991 8423 ts_tablet_manager.cc:1397] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 12:20:07.634331 8423 raft_consensus.cc:348] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.634444 8423 raft_consensus.cc:374] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.634478 8423 raft_consensus.cc:729] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.634608 8423 consensus_queue.cc:260] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.629030 8422 tablet_bootstrap.cc:654] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.634927 8423 ts_tablet_manager.cc:1428] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:20:07.636593 8421 tablet_bootstrap.cc:492] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: No bootstrap required, opened a new log
I20260620 12:20:07.636689 8421 ts_tablet_manager.cc:1397] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 12:20:07.637127 8421 raft_consensus.cc:348] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.637238 8421 raft_consensus.cc:374] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.637269 8421 raft_consensus.cc:729] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d451f69db264fad9275790a2e8d5296, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.637383 8421 consensus_queue.cc:260] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.637583 8422 tablet_bootstrap.cc:492] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: No bootstrap required, opened a new log
I20260620 12:20:07.637645 8422 ts_tablet_manager.cc:1397] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260620 12:20:07.637665 8421 ts_tablet_manager.cc:1428] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:07.637939 8422 raft_consensus.cc:348] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.638016 8422 raft_consensus.cc:374] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.638036 8422 raft_consensus.cc:729] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf046b837c28404ea4ed4e511ad1afdd, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.638168 8422 consensus_queue.cc:260] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.638409 8422 ts_tablet_manager.cc:1428] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:20:07.799342 8453 raft_consensus.cc:482] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:07.799464 8453 raft_consensus.cc:504] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.799945 8453 leader_election.cc:288] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:07.800377 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67718db09924e0388e11845d1ed95da" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf046b837c28404ea4ed4e511ad1afdd" is_pre_election: true
I20260620 12:20:07.800504 8067 raft_consensus.cc:2454] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 0.
I20260620 12:20:07.800479 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67718db09924e0388e11845d1ed95da" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296" is_pre_election: true
I20260620 12:20:07.800654 8346 raft_consensus.cc:2454] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 0.
I20260620 12:20:07.800865 8139 leader_election.cc:302] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6324bb4152364a28b7fa59276986bace, bf046b837c28404ea4ed4e511ad1afdd; no voters:
I20260620 12:20:07.801033 8453 raft_consensus.cc:2781] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:20:07.801113 8453 raft_consensus.cc:482] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:07.801160 8453 raft_consensus.cc:3037] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.802110 8453 raft_consensus.cc:504] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.802475 8453 leader_election.cc:288] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 election: Requested vote from peers bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:07.802793 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67718db09924e0388e11845d1ed95da" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf046b837c28404ea4ed4e511ad1afdd"
I20260620 12:20:07.802943 8067 raft_consensus.cc:3037] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.802906 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67718db09924e0388e11845d1ed95da" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296"
I20260620 12:20:07.803028 8346 raft_consensus.cc:3037] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:07.803864 8067 raft_consensus.cc:2454] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 1.
I20260620 12:20:07.803941 8346 raft_consensus.cc:2454] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 1.
I20260620 12:20:07.804175 8139 leader_election.cc:302] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6324bb4152364a28b7fa59276986bace, bf046b837c28404ea4ed4e511ad1afdd; no voters:
I20260620 12:20:07.804322 8453 raft_consensus.cc:2781] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:07.804405 8453 raft_consensus.cc:686] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 1 LEADER]: Becoming Leader. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Running, Role: LEADER
I20260620 12:20:07.804536 8453 consensus_queue.cc:237] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.806234 7908 catalog_manager.cc:5495] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace reported cstate change: term changed from 0 to 1, leader changed from <none> to 6324bb4152364a28b7fa59276986bace (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20260620 12:20:07.828042 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:07.825230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.0.98.2:43101)
0620 12:20:07.825357 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:07.825360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:07.825373 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:20:07.825452 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:07.825454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:07.825494 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:07.825559 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:20:07.825754 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.826248 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.827100 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:07.827345 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:07.827438 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:07.827662 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:20:07.827742 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:20:07.827810 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:07.827890 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 12:20:07.827936 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:20:07.830318 8346 raft_consensus.cc:1261] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Refusing update from remote peer 6324bb4152364a28b7fa59276986bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:07.830367 8067 raft_consensus.cc:1261] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Refusing update from remote peer 6324bb4152364a28b7fa59276986bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:07.830862 8453 consensus_queue.cc:1035] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:07.831063 8484 consensus_queue.cc:1035] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:07.839030 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:07.839143 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:07.841005 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: Sending DeleteTablet for 3 replicas of tablet f67718db09924e0388e11845d1ed95da
I20260620 12:20:07.841650 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet f67718db09924e0388e11845d1ed95da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58810
I20260620 12:20:07.841750 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet f67718db09924e0388e11845d1ed95da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37134
I20260620 12:20:07.841939 8488 tablet_replica.cc:330] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: stopping tablet replica
I20260620 12:20:07.841975 8489 tablet_replica.cc:330] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: stopping tablet replica
I20260620 12:20:07.842087 8488 raft_consensus.cc:2229] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:07.842118 8489 raft_consensus.cc:2229] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:20:07.842273 8488 raft_consensus.cc:2258] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:07.842402 8489 raft_consensus.cc:2258] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:07.842382 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 12:20:07.842459 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet f67718db09924e0388e11845d1ed95da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47632
I20260620 12:20:07.842664 8490 tablet_replica.cc:330] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: stopping tablet replica
W20260620 12:20:07.842799 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:20:07.842830 8490 raft_consensus.cc:2229] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:07.842965 8490 raft_consensus.cc:2258] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:07.843384 8488 ts_tablet_manager.cc:1905] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:07.843758 8490 ts_tablet_manager.cc:1905] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:07.843813 8489 ts_tablet_manager.cc:1905] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:07.845625 8488 ts_tablet_manager.cc:1918] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:07.845680 8488 log.cc:1196] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/f67718db09924e0388e11845d1ed95da
I20260620 12:20:07.845863 8488 ts_tablet_manager.cc:1939] T f67718db09924e0388e11845d1ed95da P bf046b837c28404ea4ed4e511ad1afdd: Deleting consensus metadata
I20260620 12:20:07.846369 8490 ts_tablet_manager.cc:1918] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:07.846364 8489 ts_tablet_manager.cc:1918] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:07.846436 8489 log.cc:1196] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/f67718db09924e0388e11845d1ed95da
I20260620 12:20:07.846436 8490 log.cc:1196] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/f67718db09924e0388e11845d1ed95da
I20260620 12:20:07.846725 8489 ts_tablet_manager.cc:1939] T f67718db09924e0388e11845d1ed95da P 6324bb4152364a28b7fa59276986bace: Deleting consensus metadata
I20260620 12:20:07.846716 8490 ts_tablet_manager.cc:1939] T f67718db09924e0388e11845d1ed95da P 5d451f69db264fad9275790a2e8d5296: Deleting consensus metadata
I20260620 12:20:07.847405 7895 catalog_manager.cc:4841] TS bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287): tablet f67718db09924e0388e11845d1ed95da (table test-table [id=016a960e173345fd8e63c419ecb866e2]) successfully deleted
I20260620 12:20:07.847608 7894 catalog_manager.cc:4841] TS 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901): tablet f67718db09924e0388e11845d1ed95da (table test-table [id=016a960e173345fd8e63c419ecb866e2]) successfully deleted
I20260620 12:20:07.848268 7894 catalog_manager.cc:4841] TS 6324bb4152364a28b7fa59276986bace (127.0.98.2:43101): tablet f67718db09924e0388e11845d1ed95da (table test-table [id=016a960e173345fd8e63c419ecb866e2]) successfully deleted
I20260620 12:20:07.850168 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 07556f30b57f435d94202df8b01840ae (DEFAULT_TABLE table=test-table [id=82211b2bac7b420abbbe90edc01ca38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:07.850242 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 07556f30b57f435d94202df8b01840ae (DEFAULT_TABLE table=test-table [id=82211b2bac7b420abbbe90edc01ca38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:07.850359 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 07556f30b57f435d94202df8b01840ae (DEFAULT_TABLE table=test-table [id=82211b2bac7b420abbbe90edc01ca38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:07.850452 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07556f30b57f435d94202df8b01840ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.850452 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07556f30b57f435d94202df8b01840ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.850548 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07556f30b57f435d94202df8b01840ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:07.852098 8421 tablet_bootstrap.cc:492] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: Bootstrap starting.
I20260620 12:20:07.852581 8422 tablet_bootstrap.cc:492] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd: Bootstrap starting.
I20260620 12:20:07.852979 8421 tablet_bootstrap.cc:654] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.853174 8423 tablet_bootstrap.cc:492] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: Bootstrap starting.
I20260620 12:20:07.853391 8422 tablet_bootstrap.cc:654] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.853883 8423 tablet_bootstrap.cc:654] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:07.854456 8421 tablet_bootstrap.cc:492] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: No bootstrap required, opened a new log
I20260620 12:20:07.854485 8422 tablet_bootstrap.cc:492] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd: No bootstrap required, opened a new log
I20260620 12:20:07.854542 8421 ts_tablet_manager.cc:1397] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:07.854557 8422 ts_tablet_manager.cc:1397] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 12:20:07.854971 8422 raft_consensus.cc:348] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.855063 8422 raft_consensus.cc:374] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.855001 8421 raft_consensus.cc:348] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.855085 8422 raft_consensus.cc:729] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf046b837c28404ea4ed4e511ad1afdd, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.855114 8421 raft_consensus.cc:374] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.855149 8421 raft_consensus.cc:729] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d451f69db264fad9275790a2e8d5296, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.855228 8422 consensus_queue.cc:260] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.855283 8421 consensus_queue.cc:260] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.855497 8422 ts_tablet_manager.cc:1428] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 12:20:07.855533 8421 ts_tablet_manager.cc:1428] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:07.856215 8423 tablet_bootstrap.cc:492] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: No bootstrap required, opened a new log
I20260620 12:20:07.856292 8423 ts_tablet_manager.cc:1397] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 12:20:07.856709 8423 raft_consensus.cc:348] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.856818 8423 raft_consensus.cc:374] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:07.856863 8423 raft_consensus.cc:729] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Initialized, Role: FOLLOWER
I20260620 12:20:07.856986 8423 consensus_queue.cc:260] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:07.857293 8423 ts_tablet_manager.cc:1428] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:20:08.196498 8484 raft_consensus.cc:482] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:08.196663 8484 raft_consensus.cc:504] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:08.197217 8484 leader_election.cc:288] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:08.197741 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07556f30b57f435d94202df8b01840ae" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf046b837c28404ea4ed4e511ad1afdd" is_pre_election: true
I20260620 12:20:08.197742 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07556f30b57f435d94202df8b01840ae" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296" is_pre_election: true
I20260620 12:20:08.197871 8067 raft_consensus.cc:2454] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 0.
I20260620 12:20:08.197911 8346 raft_consensus.cc:2454] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 0.
I20260620 12:20:08.198259 8139 leader_election.cc:302] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6324bb4152364a28b7fa59276986bace, bf046b837c28404ea4ed4e511ad1afdd; no voters:
I20260620 12:20:08.198459 8484 raft_consensus.cc:2781] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:20:08.198527 8484 raft_consensus.cc:482] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:08.198556 8484 raft_consensus.cc:3037] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:08.199589 8484 raft_consensus.cc:504] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:08.199959 8484 leader_election.cc:288] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 election: Requested vote from peers bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287), 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901)
I20260620 12:20:08.200322 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07556f30b57f435d94202df8b01840ae" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf046b837c28404ea4ed4e511ad1afdd"
I20260620 12:20:08.200321 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07556f30b57f435d94202df8b01840ae" candidate_uuid: "6324bb4152364a28b7fa59276986bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d451f69db264fad9275790a2e8d5296"
I20260620 12:20:08.200443 8067 raft_consensus.cc:3037] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:08.200443 8346 raft_consensus.cc:3037] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:08.201594 8067 raft_consensus.cc:2454] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 1.
I20260620 12:20:08.201594 8346 raft_consensus.cc:2454] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6324bb4152364a28b7fa59276986bace in term 1.
I20260620 12:20:08.201881 8138 leader_election.cc:302] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d451f69db264fad9275790a2e8d5296, 6324bb4152364a28b7fa59276986bace; no voters:
I20260620 12:20:08.202071 8484 raft_consensus.cc:2781] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:08.202193 8484 raft_consensus.cc:686] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 1 LEADER]: Becoming Leader. State: Replica: 6324bb4152364a28b7fa59276986bace, State: Running, Role: LEADER
I20260620 12:20:08.202328 8484 consensus_queue.cc:237] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } }
I20260620 12:20:08.203936 7910 catalog_manager.cc:5495] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace reported cstate change: term changed from 0 to 1, leader changed from <none> to 6324bb4152364a28b7fa59276986bace (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6324bb4152364a28b7fa59276986bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6324bb4152364a28b7fa59276986bace" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20260620 12:20:08.229606 8346 raft_consensus.cc:1261] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Refusing update from remote peer 6324bb4152364a28b7fa59276986bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:08.229712 8067 raft_consensus.cc:1261] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Refusing update from remote peer 6324bb4152364a28b7fa59276986bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:08.229982 8427 consensus_queue.cc:1035] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d451f69db264fad9275790a2e8d5296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:08.230122 8484 consensus_queue.cc:1035] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf046b837c28404ea4ed4e511ad1afdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:08.272755 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:08.272898 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:08.274827 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54fbd9ece10546ddb4eb2436f65ef388: Sending DeleteTablet for 3 replicas of tablet 07556f30b57f435d94202df8b01840ae
I20260620 12:20:08.275375 8187 tablet_service.cc:1514] Processing DeleteTablet for tablet 07556f30b57f435d94202df8b01840ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37134
I20260620 12:20:08.275357 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 07556f30b57f435d94202df8b01840ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58810
I20260620 12:20:08.275537 8488 tablet_replica.cc:330] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd: stopping tablet replica
I20260620 12:20:08.275540 8489 tablet_replica.cc:330] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: stopping tablet replica
I20260620 12:20:08.275516 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 07556f30b57f435d94202df8b01840ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47632
I20260620 12:20:08.275642 8488 raft_consensus.cc:2229] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:08.275657 8489 raft_consensus.cc:2229] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:20:08.275708 8490 tablet_replica.cc:330] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: stopping tablet replica
I20260620 12:20:08.275753 8488 raft_consensus.cc:2258] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:08.275825 8490 raft_consensus.cc:2229] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:08.275831 8489 raft_consensus.cc:2258] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:08.275926 8490 raft_consensus.cc:2258] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:08.276417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 7982
I20260620 12:20:08.276458 8488 ts_tablet_manager.cc:1905] T 07556f30b57f435d94202df8b01840ae P bf046b837c28404ea4ed4e511ad1afdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:08.276556 8490 ts_tablet_manager.cc:1905] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:08.278720 8490 ts_tablet_manager.cc:1918] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:20:08.278824 8490 log.cc:1196] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/07556f30b57f435d94202df8b01840ae
I20260620 12:20:08.279086 8490 ts_tablet_manager.cc:1939] T 07556f30b57f435d94202df8b01840ae P 5d451f69db264fad9275790a2e8d5296: Deleting consensus metadata
I20260620 12:20:08.279191 8489 ts_tablet_manager.cc:1905] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:08.279798 7894 catalog_manager.cc:4841] TS 5d451f69db264fad9275790a2e8d5296 (127.0.98.3:36901): tablet 07556f30b57f435d94202df8b01840ae (table test-table [id=82211b2bac7b420abbbe90edc01ca38e]) successfully deleted
I20260620 12:20:08.281265 8489 ts_tablet_manager.cc:1918] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 12:20:08.281327 8489 log.cc:1196] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/07556f30b57f435d94202df8b01840ae
I20260620 12:20:08.281500 8489 ts_tablet_manager.cc:1939] T 07556f30b57f435d94202df8b01840ae P 6324bb4152364a28b7fa59276986bace: Deleting consensus metadata
I20260620 12:20:08.282058 7894 catalog_manager.cc:4841] TS 6324bb4152364a28b7fa59276986bace (127.0.98.2:43101): tablet 07556f30b57f435d94202df8b01840ae (table test-table [id=82211b2bac7b420abbbe90edc01ca38e]) successfully deleted
W20260620 12:20:08.284096 7895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39287 (error 108)
I20260620 12:20:08.284880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 8122
I20260620 12:20:08.285181 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:08.284755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39287 (local address 127.0.0.1:57920)
0620 12:20:08.284980 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:08.285072 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
W20260620 12:20:08.285833 7895 catalog_manager.cc:4584] TS bf046b837c28404ea4ed4e511ad1afdd (127.0.98.1:39287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07556f30b57f435d94202df8b01840ae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39287: connect: Connection refused (error 111)
I20260620 12:20:08.291996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 8262
I20260620 12:20:08.297541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 7878
2026-06-20T12:20:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3735 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:20:08 dist-test-slave-9b3m krb5kdc[8522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:20:08 dist-test-slave-9b3m krb5kdc[8522](info): set up 2 sockets
Jun 20 12:20:08 dist-test-slave-9b3m krb5kdc[8522](info): commencing operation
krb5kdc: starting...
W20260620 12:20:09.397806 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:20:09 dist-test-slave-9b3m krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:20:09.519795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:09.630131 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:09.630455 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:09.630533 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:09.634763 8535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:20:09.634846 8535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:09.634883 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:09.634939 8535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:20:09.634979 8535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:20:09.640323 8535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:09.641408 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:09.642450 8535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:09.651149 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:09.651244 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:09.651450 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:09.651312 8535 server_base.cc:1034] running on GCE node
I20260620 12:20:09.652169 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:09.653577 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:09.654779 8535 hybrid_clock.cc:648] HybridClock initialized: now 1781958009654763 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T12:20:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1369 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:20:09 dist-test-slave-9b3m krb5kdc[8548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:20:09 dist-test-slave-9b3m krb5kdc[8548](info): set up 2 sockets
Jun 20 12:20:09 dist-test-slave-9b3m krb5kdc[8548](info): commencing operation
krb5kdc: starting...
W20260620 12:20:10.745864 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:20:10 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:20:10.864022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45433
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:10.972851 8561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:10.973183 8561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:10.973263 8561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:10.977223 8561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:20:10.977314 8561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:10.977350 8561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:10.977416 8561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:20:10.977473 8561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:20:10.982748 8561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:10.983820 8561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:10.984880 8561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:10.993359 8569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:10.993359 8566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:10.993386 8567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:10.993585 8561 server_base.cc:1034] running on GCE node
I20260620 12:20:10.994262 8561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:10.995491 8561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:10.996706 8561 hybrid_clock.cc:648] HybridClock initialized: now 1781958010996694 us; error 30 us; skew 500 ppm
Jun 20 12:20:10 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:10.999675 8561 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:20:11.000660 8561 webserver.cc:460] Webserver started at http://127.0.98.62:34377/ using document root <none> and password file <none>
I20260620 12:20:11.001323 8561 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:11.001420 8561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:11.001673 8561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:11.003856 8561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "16fa7288ebe541e386a390079523a4c3"
format_stamp: "Formatted at 2026-06-20 12:20:10 on dist-test-slave-9b3m"
server_key: "050838ec08f79c6a75b65e64ff00b550"
server_key_iv: "afbd9314e5c7adc144936cc04d84154d"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.004455 8561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "16fa7288ebe541e386a390079523a4c3"
format_stamp: "Formatted at 2026-06-20 12:20:10 on dist-test-slave-9b3m"
server_key: "050838ec08f79c6a75b65e64ff00b550"
server_key_iv: "afbd9314e5c7adc144936cc04d84154d"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.008327 8561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:20:11.010995 8576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.012281 8561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:11.012428 8561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "16fa7288ebe541e386a390079523a4c3"
format_stamp: "Formatted at 2026-06-20 12:20:10 on dist-test-slave-9b3m"
server_key: "050838ec08f79c6a75b65e64ff00b550"
server_key_iv: "afbd9314e5c7adc144936cc04d84154d"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.012548 8561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:11.028605 8561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:11.031616 8561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:11.031821 8561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:11.039121 8561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45433
I20260620 12:20:11.039148 8628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45433 every 8 connection(s)
I20260620 12:20:11.040096 8561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:20:11.042915 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:11.048693 8629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Bootstrap starting.
I20260620 12:20:11.049934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 8561
I20260620 12:20:11.050102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:20:11.050414 392 external_mini_cluster.cc:1428] Setting key 2f2212c622ddb6405f9c744ed52a9f7a
I20260620 12:20:11.051611 8629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:11.052589 8629 log.cc:824] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:11.054890 8629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: No bootstrap required, opened a new log
I20260620 12:20:11.059171 8629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260620 12:20:11.059420 8629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:11.059495 8629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fa7288ebe541e386a390079523a4c3, State: Initialized, Role: FOLLOWER
Jun 20 12:20:11 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:20:11.060020 8629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260620 12:20:11.060184 8629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:20:11.060262 8629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:20:11.060371 8629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:11.061405 8629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260620 12:20:11.061760 8629 leader_election.cc:302] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16fa7288ebe541e386a390079523a4c3; no voters:
I20260620 12:20:11.062090 8629 leader_election.cc:288] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:20:11.062162 8634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:11.062361 8634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 16fa7288ebe541e386a390079523a4c3, State: Running, Role: LEADER
I20260620 12:20:11.062709 8634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260620 12:20:11.063385 8629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:20:11.064369 8636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16fa7288ebe541e386a390079523a4c3. Latest consensus state: current_term: 1 leader_uuid: "16fa7288ebe541e386a390079523a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } } }
I20260620 12:20:11.064777 8636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:11.064663 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16fa7288ebe541e386a390079523a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } } }
I20260620 12:20:11.065117 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:11.065675 8643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:20:11.066455 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:11.051553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.0.98.62:45433)
0620 12:20:11.052060 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:11.052080 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:11.052124 (+ 44us) server_negotiation.cc:407] Connection header received
0620 12:20:11.053017 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:11.053093 (+ 76us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:11.053482 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:11.053859 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:11.054525 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.056098 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.056937 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.057762 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.057967 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:11.060786 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:11.060790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:11.060802 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:11.060826 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:11.062968 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:11.063427 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:11.063433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:11.063438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:11.063567 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:11.063877 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:11.063880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:11.063882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:11.064382 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:11.064568 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:11.065221 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 12:20:11.065512 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":148,"threads_started":1}
I20260620 12:20:11.068826 8643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:20:11.074471 8643 catalog_manager.cc:1347] Generated new cluster ID: 9b7fa780dea84e7199dd39c11a84782e
I20260620 12:20:11.074572 8643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:20:11.083817 8643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:20:11.085057 8643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:20:11.094398 8643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Generated new TSK 0
I20260620 12:20:11.095117 8643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:20:11.133164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45433
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:11.245621 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:11.245880 8655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:11.245923 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:11.250010 8655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:11.250066 8655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:11.250154 8655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:20:11.255465 8655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:11.256527 8655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:11.257632 8655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:11.267549 8660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:11.267601 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:11.267597 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:11.268033 8655 server_base.cc:1034] running on GCE node
I20260620 12:20:11.268445 8655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:11.269193 8655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:11.270406 8655 hybrid_clock.cc:648] HybridClock initialized: now 1781958011270365 us; error 51 us; skew 500 ppm
Jun 20 12:20:11 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:11.273604 8655 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:20:11.274554 8655 webserver.cc:460] Webserver started at http://127.0.98.1:46479/ using document root <none> and password file <none>
I20260620 12:20:11.275134 8655 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:11.275226 8655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:11.275467 8655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:11.277601 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "a1d857bcba9d49c4a69ec1347235d1eb"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "7528ebac702d1aeca289d76b253cf1a9"
server_key_iv: "2fd595f9360b77ba9dda8aba79e4b71d"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.278151 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a1d857bcba9d49c4a69ec1347235d1eb"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "7528ebac702d1aeca289d76b253cf1a9"
server_key_iv: "2fd595f9360b77ba9dda8aba79e4b71d"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.281996 8655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:20:11.284546 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.285939 8655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:20:11.286087 8655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "a1d857bcba9d49c4a69ec1347235d1eb"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "7528ebac702d1aeca289d76b253cf1a9"
server_key_iv: "2fd595f9360b77ba9dda8aba79e4b71d"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.286202 8655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:11.304647 8655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:11.308080 8655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:11.308290 8655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:11.308957 8655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:11.310017 8655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:11.310089 8655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.310156 8655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:11.310205 8655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.319420 8655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42121
I20260620 12:20:11.319440 8783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42121 every 8 connection(s)
I20260620 12:20:11.320394 8655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:20:11.330066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 8655
Jun 20 12:20:11 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:20:11.330186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:20:11.330526 392 external_mini_cluster.cc:1428] Setting key 5f02c1865a0730c688a3fd410f16db83
I20260620 12:20:11.334961 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:11.322323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39447 (local address 127.0.98.62:45433)
0620 12:20:11.322479 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:11.322483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:11.323319 (+ 836us) server_negotiation.cc:407] Connection header received
0620 12:20:11.324709 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:11.324712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:11.324766 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:11.324854 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:11.326061 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.326594 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.327713 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.327958 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.328070 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:11.330924 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:11.330932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:11.330937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:11.330967 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:11.332711 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:11.333349 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:11.333352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:11.333354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:11.333439 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:11.333797 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:11.333800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:11.333802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:11.334025 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:11.334134 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:11.334656 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 12:20:11.334779 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:20:11.335757 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:11.322709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:39447)
0620 12:20:11.323164 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:11.323206 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:11.324500 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:11.325025 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:11.325032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:11.325446 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:11.325914 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:11.325927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.326717 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.326720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:11.327585 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:11.327594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.328109 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.328113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:11.328276 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:11.328888 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:11.328904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:11.330698 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:11.332868 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:11.332877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:11.332893 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:11.333234 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:11.333546 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:11.333549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:11.333551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:11.333694 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:11.334145 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:11.334153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:11.334413 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 12:20:11.334602 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 12:20:11.334932 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260620 12:20:11.336967 8784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45433
I20260620 12:20:11.337257 8784 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:11.337869 8784 heartbeater.cc:511] Master 127.0.98.62:45433 requested a full tablet report, sending...
I20260620 12:20:11.339748 8593 ts_manager.cc:194] Registered new tserver with Master: a1d857bcba9d49c4a69ec1347235d1eb (127.0.98.1:42121)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:20:11.341022 8593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:20:11.360435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45433
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:11.471091 8789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:11.471340 8789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:11.471385 8789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:11.475446 8789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:11.475500 8789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:11.475598 8789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:20:11.480794 8789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:11.481837 8789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:11.482779 8789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:11.492370 8794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:11.492377 8795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:11.492393 8797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:11.492614 8789 server_base.cc:1034] running on GCE node
I20260620 12:20:11.493160 8789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:11.493786 8789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:11.494969 8789 hybrid_clock.cc:648] HybridClock initialized: now 1781958011494956 us; error 25 us; skew 500 ppm
Jun 20 12:20:11 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:11.497641 8789 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:20:11.498545 8789 webserver.cc:460] Webserver started at http://127.0.98.2:36011/ using document root <none> and password file <none>
I20260620 12:20:11.499096 8789 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:11.499185 8789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:11.499409 8789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:11.501456 8789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "16a8b70d95f84de3893729c62c73d4e8"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "ac367ab75257e50ba02e3448d1eaf283"
server_key_iv: "203f8df0bcc34c8c9ba068d3d3b1bb9a"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.501998 8789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "16a8b70d95f84de3893729c62c73d4e8"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "ac367ab75257e50ba02e3448d1eaf283"
server_key_iv: "203f8df0bcc34c8c9ba068d3d3b1bb9a"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.505501 8789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:20:11.507895 8804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.509004 8789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:20:11.509152 8789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "16a8b70d95f84de3893729c62c73d4e8"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "ac367ab75257e50ba02e3448d1eaf283"
server_key_iv: "203f8df0bcc34c8c9ba068d3d3b1bb9a"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.509256 8789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:11.521150 8789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:11.523667 8789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:11.523864 8789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:11.524405 8789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:11.525252 8789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:11.525318 8789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.525386 8789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:11.525429 8789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.534466 8789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42557
I20260620 12:20:11.534492 8917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42557 every 8 connection(s)
I20260620 12:20:11.535307 8789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:20:11.536827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 8789
I20260620 12:20:11.536964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:20:11.537360 392 external_mini_cluster.cc:1428] Setting key 861c509d787dcf218a041e62fbc0d8a9
Jun 20 12:20:11 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:20:11.550619 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:11.537175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50057 (local address 127.0.98.62:45433)
0620 12:20:11.537333 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:11.537339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:11.538352 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 12:20:11.539919 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:11.539925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:11.539985 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:11.540081 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:11.541426 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.542006 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.543239 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.543493 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.543609 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:11.546379 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:11.546386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:11.546391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:11.546422 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:11.548053 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:11.548765 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:11.548768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:11.548770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:11.548937 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:11.549348 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:11.549357 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:11.549359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:11.549593 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:11.549705 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:11.550199 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 12:20:11.550338 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:20:11.551362 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:11.537675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:50057)
0620 12:20:11.538192 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:11.538245 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:11.539717 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:11.540266 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:11.540274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:11.540647 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:11.541264 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:11.541280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.542141 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.542145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:11.543104 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:11.543114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.543659 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.543664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:11.543838 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:11.544475 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:11.544490 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:11.546187 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:11.548324 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:11.548332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:11.548347 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:11.548642 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:11.549066 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:11.549072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:11.549077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:11.549250 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:11.549709 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:11.549714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:11.549935 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 12:20:11.550102 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 12:20:11.550342 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260620 12:20:11.552671 8918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45433
I20260620 12:20:11.552937 8918 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:11.553524 8918 heartbeater.cc:511] Master 127.0.98.62:45433 requested a full tablet report, sending...
I20260620 12:20:11.554687 8593 ts_manager.cc:194] Registered new tserver with Master: 16a8b70d95f84de3893729c62c73d4e8 (127.0.98.2:42557)
I20260620 12:20:11.555488 8593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:20:11.566274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45433
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:11.677944 8923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:11.678177 8923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:11.678218 8923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:11.682224 8923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:11.682276 8923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:11.682332 8923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:20:11.687389 8923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:11.688310 8923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:11.689416 8923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:20:11.698848 8923 server_base.cc:1034] running on GCE node
W20260620 12:20:11.698814 8929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:11.698825 8928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:11.698990 8931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:11.699400 8923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:11.699983 8923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:11.701171 8923 hybrid_clock.cc:648] HybridClock initialized: now 1781958011701152 us; error 30 us; skew 500 ppm
Jun 20 12:20:11 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:11.703804 8923 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:20:11.704641 8923 webserver.cc:460] Webserver started at http://127.0.98.3:34861/ using document root <none> and password file <none>
I20260620 12:20:11.705248 8923 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:11.705338 8923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:11.705557 8923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:11.707515 8923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d68485df8774c67b9f493e97401a70e"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "4396ac3d285f0a6f6fcbfb727c236a1e"
server_key_iv: "ae703a349f2864542555c4f42003fc8c"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.708055 8923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d68485df8774c67b9f493e97401a70e"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "4396ac3d285f0a6f6fcbfb727c236a1e"
server_key_iv: "ae703a349f2864542555c4f42003fc8c"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.711601 8923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:20:11.714150 8938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.715354 8923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:20:11.715504 8923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "9d68485df8774c67b9f493e97401a70e"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "4396ac3d285f0a6f6fcbfb727c236a1e"
server_key_iv: "ae703a349f2864542555c4f42003fc8c"
server_key_version: "encryptionkey@0"
I20260620 12:20:11.715615 8923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:11.725418 8923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:11.728600 8923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:11.728802 8923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:11.729432 8923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:11.730248 8923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:11.730326 8923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.730402 8923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:11.730455 8923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:11.739385 8923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46325
I20260620 12:20:11.739411 9051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46325 every 8 connection(s)
I20260620 12:20:11.740322 8923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:20:11.742391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 8923
I20260620 12:20:11.742532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:20:11.742810 392 external_mini_cluster.cc:1428] Setting key 69bc86170275204545e1d15856094034
Jun 20 12:20:11 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:20:11.753310 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:11.742066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52627 (local address 127.0.98.62:45433)
0620 12:20:11.742230 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:11.742235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:11.743164 (+ 929us) server_negotiation.cc:407] Connection header received
0620 12:20:11.744525 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:11.744528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:11.744582 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:11.744671 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:11.745938 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.746455 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.747592 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.747801 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.747880 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:11.750319 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:11.750322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:11.750325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:11.750346 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:11.751538 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:11.751983 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:11.751986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:11.751987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:11.752054 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:11.752320 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:11.752322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:11.752323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:11.752483 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:11.752557 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:11.753068 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 12:20:11.753197 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:20:11.754086 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:11.742455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:52627)
0620 12:20:11.743022 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:11.743071 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:11.744337 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:11.744897 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:11.744905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:11.745336 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:11.745810 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:11.745824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.746598 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.746601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:11.747480 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:11.747489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:11.747935 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:11.747938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:11.748070 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:11.748680 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:11.748696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:11.750179 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:11.751631 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:11.751635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:11.751645 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:11.751897 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:11.752125 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:11.752127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:11.752129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:11.752246 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:11.752594 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:11.752599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:11.752809 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 12:20:11.753003 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 12:20:11.753309 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":158,"threads_started":1}
I20260620 12:20:11.755280 9052 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45433
I20260620 12:20:11.755520 9052 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:11.755982 9052 heartbeater.cc:511] Master 127.0.98.62:45433 requested a full tablet report, sending...
I20260620 12:20:11.756942 8593 ts_manager.cc:194] Registered new tserver with Master: 9d68485df8774c67b9f493e97401a70e (127.0.98.3:46325)
I20260620 12:20:11.757503 8593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52627
I20260620 12:20:11.767997 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 12:20:11.839970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 8655
I20260620 12:20:11.845240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 8789
I20260620 12:20:11.850517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 8923
I20260620 12:20:11.855614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 8561
I20260620 12:20:11.861673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45433
--webserver_interface=127.0.98.62
--webserver_port=34377
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:11.968628 9061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:11.968924 9061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:11.968997 9061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:11.972918 9061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:20:11.973013 9061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:11.973053 9061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:11.973124 9061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:20:11.973161 9061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:20:11.978498 9061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:11.979521 9061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:11.980499 9061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:11.988965 9066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:11.989118 9067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:11.988965 9069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:11.989557 9061 server_base.cc:1034] running on GCE node
I20260620 12:20:11.990064 9061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:11.991088 9061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:11.992285 9061 hybrid_clock.cc:648] HybridClock initialized: now 1781958011992269 us; error 36 us; skew 500 ppm
Jun 20 12:20:11 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:11.995083 9061 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:20:11.995973 9061 webserver.cc:460] Webserver started at http://127.0.98.62:34377/ using document root <none> and password file <none>
I20260620 12:20:11.996522 9061 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:11.996634 9061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:12.000326 9061 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:12.002440 9076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.003711 9061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:12.003887 9061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "16fa7288ebe541e386a390079523a4c3"
format_stamp: "Formatted at 2026-06-20 12:20:10 on dist-test-slave-9b3m"
server_key: "050838ec08f79c6a75b65e64ff00b550"
server_key_iv: "afbd9314e5c7adc144936cc04d84154d"
server_key_version: "encryptionkey@0"
I20260620 12:20:12.004401 9061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:12.018002 9061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:12.021036 9061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:12.021306 9061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:12.028570 9061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45433
I20260620 12:20:12.028591 9128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45433 every 8 connection(s)
I20260620 12:20:12.029699 9061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:20:12.033774 9129 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:20:12.035297 9129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Bootstrap starting.
I20260620 12:20:12.037828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 9061
I20260620 12:20:12.038424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42121
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45433
--webserver_port=46479
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:20:12.054062 9129 log.cc:824] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:12.060504 9129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:20:12.060866 9129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Bootstrap complete.
I20260620 12:20:12.064961 9129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260620 12:20:12.065395 9129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fa7288ebe541e386a390079523a4c3, State: Initialized, Role: FOLLOWER
I20260620 12:20:12.065867 9129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260620 12:20:12.065995 9129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:20:12.066082 9129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:20:12.066177 9129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:20:12.068010 9129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260620 12:20:12.068329 9129 leader_election.cc:302] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16fa7288ebe541e386a390079523a4c3; no voters:
I20260620 12:20:12.068683 9129 leader_election.cc:288] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:20:12.068776 9133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:20:12.068961 9133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [term 2 LEADER]: Becoming Leader. State: Replica: 16fa7288ebe541e386a390079523a4c3, State: Running, Role: LEADER
I20260620 12:20:12.069257 9133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260620 12:20:12.069886 9129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:20:12.070112 9134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16fa7288ebe541e386a390079523a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } } }
I20260620 12:20:12.070129 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16fa7288ebe541e386a390079523a4c3. Latest consensus state: current_term: 2 leader_uuid: "16fa7288ebe541e386a390079523a4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fa7288ebe541e386a390079523a4c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } } }
I20260620 12:20:12.070405 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:12.070629 9134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:12.071837 9137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:20:12.075176 9137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:20:12.075788 9137 catalog_manager.cc:1259] Loaded cluster ID: 9b7fa780dea84e7199dd39c11a84782e
I20260620 12:20:12.075850 9137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:20:12.078150 9137 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:20:12.078780 9137 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16fa7288ebe541e386a390079523a4c3: Loaded TSK: 0
I20260620 12:20:12.079943 9137 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:20:12.152129 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:12.152417 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:12.152493 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:12.156626 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:12.156704 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:12.156791 9131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:20:12.162050 9131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46479
--tserver_master_addrs=127.0.98.62:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:12.163030 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:12.163996 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:12.174548 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:12.174535 9155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:12.174726 9131 server_base.cc:1034] running on GCE node
W20260620 12:20:12.174551 9158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:12.175464 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:12.176218 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:12.177424 9131 hybrid_clock.cc:648] HybridClock initialized: now 1781958012177395 us; error 50 us; skew 500 ppm
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958012, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:12.180775 9131 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 12:20:12.181721 9131 webserver.cc:460] Webserver started at http://127.0.98.1:46479/ using document root <none> and password file <none>
I20260620 12:20:12.182310 9131 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:12.182384 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:12.186501 9131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:20:12.188606 9165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.190056 9131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:20:12.190229 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "a1d857bcba9d49c4a69ec1347235d1eb"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "7528ebac702d1aeca289d76b253cf1a9"
server_key_iv: "2fd595f9360b77ba9dda8aba79e4b71d"
server_key_version: "encryptionkey@0"
I20260620 12:20:12.190827 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:12.222558 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:12.226106 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:12.226323 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:12.226955 9131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:12.227820 9131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:12.227897 9131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.227972 9131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:12.228031 9131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.237457 9131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42121
I20260620 12:20:12.237504 9278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42121 every 8 connection(s)
I20260620 12:20:12.238528 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:20:12.246393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 9131
I20260620 12:20:12.246853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42557
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45433
--webserver_port=36011
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:20:12.253031 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.240791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44585 (local address 127.0.98.62:45433)
0620 12:20:12.241298 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.241319 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.241573 (+ 254us) server_negotiation.cc:407] Connection header received
0620 12:20:12.243072 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.243082 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.243367 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.243644 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.244756 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.245792 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.247238 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.248044 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.248237 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.252385 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44585: BlockingRecv error: recv got EOF from 127.0.98.1:44585 (error 108)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":112,"threads_started":1}
I20260620 12:20:12.253161 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.240917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:44585)
0620 12:20:12.241420 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.241460 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.242645 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.243748 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.243754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.244138 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.244584 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.244597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.245962 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.245966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.247111 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.247122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.248258 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.248264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.248490 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.249247 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.249272 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.252092 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":171,"threads_started":1}
W20260620 12:20:12.253844 9279 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45433 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45433: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:12.259332 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.254342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:58889)
0620 12:20:12.254461 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.254477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.254559 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.254861 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.254864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.255021 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.255106 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.255111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.255834 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.255837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.256386 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.256394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.256828 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.256832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.256926 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.257454 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.257466 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.259213 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 12:20:12.259720 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.254416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58889 (local address 127.0.98.62:45433)
0620 12:20:12.254532 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.254535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.254548 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:20:12.254680 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.254682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.254721 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.254841 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.255217 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.255706 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.256503 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.256718 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.256828 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.259366 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58889: BlockingRecv error: recv got EOF from 127.0.98.1:58889 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:12.265218 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.259821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:50531)
0620 12:20:12.259971 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.259984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.260046 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.260271 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.260274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.260426 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.260505 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.260511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.261329 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.261332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.262185 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.262196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.262668 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.262672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.262750 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.263198 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.263211 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.265108 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 12:20:12.265407 9279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:20:12.265620 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.259921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50531 (local address 127.0.98.62:45433)
0620 12:20:12.260041 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.260046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.260060 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:12.260133 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.260135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.260172 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.260235 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.260610 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.261226 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.262314 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.262558 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.262651 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.265242 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50531: BlockingRecv error: recv got EOF from 127.0.98.1:50531 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 12:20:12.358939 9283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:12.359229 9283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:12.359305 9283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:12.363335 9283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:12.363415 9283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:12.363503 9283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:20:12.368531 9283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36011
--tserver_master_addrs=127.0.98.62:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:12.369736 9283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:12.370807 9283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:12.380971 9289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:12.380983 9291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:12.380992 9288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:12.381302 9283 server_base.cc:1034] running on GCE node
I20260620 12:20:12.381678 9283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:12.382272 9283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:12.383445 9283 hybrid_clock.cc:648] HybridClock initialized: now 1781958012383411 us; error 47 us; skew 500 ppm
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958012, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:12.386330 9283 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 12:20:12.387141 9283 webserver.cc:460] Webserver started at http://127.0.98.2:36011/ using document root <none> and password file <none>
I20260620 12:20:12.387696 9283 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:12.387764 9283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:12.391639 9283 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:12.393710 9298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.394914 9283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:12.395015 9283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "16a8b70d95f84de3893729c62c73d4e8"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "ac367ab75257e50ba02e3448d1eaf283"
server_key_iv: "203f8df0bcc34c8c9ba068d3d3b1bb9a"
server_key_version: "encryptionkey@0"
I20260620 12:20:12.395573 9283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:12.406018 9283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:12.409210 9283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:12.409354 9283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:12.409857 9283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:12.410622 9283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:12.410669 9283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.410709 9283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:12.410725 9283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.419145 9283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42557
I20260620 12:20:12.419212 9411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42557 every 8 connection(s)
I20260620 12:20:12.420153 9283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:20:12.423579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 9283
I20260620 12:20:12.423996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46325
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45433
--webserver_port=34861
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:20:12.432322 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.422072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46555 (local address 127.0.98.62:45433)
0620 12:20:12.422237 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.422242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.423025 (+ 783us) server_negotiation.cc:407] Connection header received
0620 12:20:12.424530 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.424536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.424605 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.424705 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.426267 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.426778 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.427983 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.428217 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.428327 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.432182 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46555: BlockingRecv error: recv got EOF from 127.0.98.2:46555 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:20:12.432961 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.422428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:46555)
0620 12:20:12.422864 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.422906 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.424316 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.424859 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.424865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.425374 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.426099 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.426120 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.426918 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.426921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.427851 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.427860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.428384 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.428387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.428545 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.429158 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.429173 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.432058 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
W20260620 12:20:12.433482 9412 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45433 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45433: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:12.439452 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.433925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:53679)
0620 12:20:12.434038 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.434054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.434140 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.434416 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.434418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.434551 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.434641 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.434645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.435680 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.435683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.436350 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.436359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.436935 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.436938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.437020 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.437570 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.437581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.439363 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 12:20:12.439636 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.434015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53679 (local address 127.0.98.62:45433)
0620 12:20:12.434152 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.434159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.434176 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:20:12.434224 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.434228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.434283 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.434386 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.434759 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.435551 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.436469 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.436813 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.436913 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.439516 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53679: BlockingRecv error: recv got EOF from 127.0.98.2:53679 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:12.444969 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.440034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:43981)
0620 12:20:12.440178 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.440193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.440313 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.440564 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.440566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.440688 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.440755 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.440759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.441613 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.441619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.442124 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.442131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.442557 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.442561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.442634 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.443121 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.443132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.444835 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":87,"threads_started":1}
W20260620 12:20:12.445130 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.440091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43981 (local address 127.0.98.62:45433)
0620 12:20:12.440193 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.440198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.440248 (+ 50us) server_negotiation.cc:407] Connection header received
0620 12:20:12.440405 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.440408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.440459 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.440543 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.440950 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.441504 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.442238 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.442452 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.442557 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.444995 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43981: BlockingRecv error: recv got EOF from 127.0.98.2:43981 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 12:20:12.445294 9412 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:20:12.538190 9415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:12.538487 9415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:12.538578 9415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:12.542771 9415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:12.542857 9415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:12.542949 9415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:20:12.548089 9415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34861
--tserver_master_addrs=127.0.98.62:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:12.549198 9415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:12.550256 9415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:12.560520 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:12.560572 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:12.560583 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:12.560767 9415 server_base.cc:1034] running on GCE node
I20260620 12:20:12.561381 9415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:12.562059 9415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:12.563274 9415 hybrid_clock.cc:648] HybridClock initialized: now 1781958012563256 us; error 32 us; skew 500 ppm
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958012, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:12.566126 9415 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 12:20:12.566977 9415 webserver.cc:460] Webserver started at http://127.0.98.3:34861/ using document root <none> and password file <none>
I20260620 12:20:12.567531 9415 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:12.567636 9415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:12.571883 9415 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 12:20:12.573951 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.575176 9415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:20:12.575304 9415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "9d68485df8774c67b9f493e97401a70e"
format_stamp: "Formatted at 2026-06-20 12:20:11 on dist-test-slave-9b3m"
server_key: "4396ac3d285f0a6f6fcbfb727c236a1e"
server_key_iv: "ae703a349f2864542555c4f42003fc8c"
server_key_version: "encryptionkey@0"
I20260620 12:20:12.575827 9415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:12.600047 9415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:12.603003 9415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:12.603186 9415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:12.603747 9415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:12.604537 9415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:12.604605 9415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.604672 9415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:12.604727 9415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:12.614094 9415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46325
I20260620 12:20:12.614120 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46325 every 8 connection(s)
I20260620 12:20:12.615020 9415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:20:12.620800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 9415
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:20:12.626570 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.616734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46309 (local address 127.0.98.62:45433)
0620 12:20:12.616911 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.616914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.617657 (+ 743us) server_negotiation.cc:407] Connection header received
0620 12:20:12.618962 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.618966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.619019 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.619098 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.620167 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.620690 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.622276 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.622510 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.622633 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.626455 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46309: BlockingRecv error: recv got EOF from 127.0.98.3:46309 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:20:12.627177 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.617123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:46309)
0620 12:20:12.617517 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.617557 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.618755 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.619252 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.619258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.619583 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.620028 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.620041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.620819 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.620824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.622158 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.622167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.622653 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.622656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.622837 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.623450 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.623471 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.626357 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
W20260620 12:20:12.627729 9545 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45433 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45433: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:12.633440 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.621910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.0.98.62:45433)
0620 12:20:12.622193 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.622197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.622221 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:20:12.622252 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.622254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.622291 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.622399 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.623224 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.624036 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.625455 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.625807 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.625925 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.628819 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:12.628825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:12.628843 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:12.628874 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:12.630898 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:12.631415 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:12.631420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:12.631425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:12.631593 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:12.631964 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:12.631967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:12.631969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:12.632444 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:12.632649 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:12.633023 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 12:20:12.633281 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:12.634159 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.628213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:35789)
0620 12:20:12.628323 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.628341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.628423 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.628672 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.628675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.628849 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.628939 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.628944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.629964 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.629968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.630883 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.630893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.631506 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.631510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.631600 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.632076 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.632089 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.634028 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 12:20:12.635795 9281 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.628315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35789 (local address 127.0.98.62:45433)
0620 12:20:12.628445 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.628452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.628475 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:20:12.628516 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.628519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.628556 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.628658 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.629082 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.629842 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.631025 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.631393 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.631492 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.635700 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35789: BlockingRecv error: recv got EOF from 127.0.98.3:35789 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:12 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:12.642061 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:12.634909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:40457)
0620 12:20:12.635386 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:12.635402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:12.635494 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:12.636059 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:12.636062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:12.636223 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:12.636314 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.636320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.637524 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.637528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.638457 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:12.638469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.639123 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.639126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:12.639220 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.639711 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:12.639724 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:12.641906 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397}
W20260620 12:20:12.642160 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:12.634986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40457 (local address 127.0.98.62:45433)
0620 12:20:12.635136 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:12.635141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:12.635473 (+ 332us) server_negotiation.cc:407] Connection header received
0620 12:20:12.635836 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:12.635840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:12.635884 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:12.635960 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:12.636436 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.637396 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.638594 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:12.639007 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:12.639130 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:12.642063 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40457: BlockingRecv error: recv got EOF from 127.0.98.3:40457 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 12:20:12.642319 9545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 12:20:13 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:13 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:13.273593 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:13.266082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:34789)
0620 12:20:13.266362 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:13.266380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:13.266480 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:13.266820 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:13.266825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:13.267082 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:13.267246 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:13.267257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.268582 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.268585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:13.269644 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:13.269657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.270132 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.270136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:13.270236 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:13.270959 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:13.270973 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:13.273393 (+ 2420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
W20260620 12:20:13.273667 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:13.266158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34789 (local address 127.0.98.62:45433)
0620 12:20:13.266448 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:13.266452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:13.266467 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:20:13.266592 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:13.266596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:13.266675 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:13.266797 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:13.267396 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.268451 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.269762 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.270009 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.270136 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:13.273522 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34789: BlockingRecv error: recv got EOF from 127.0.98.1:34789 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":122,"threads_started":1}
Jun 20 12:20:13 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:13 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:13.455574 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:13.445971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:48185)
0620 12:20:13.446283 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:13.446297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:13.446401 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:13.446624 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:13.446626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:13.446841 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:13.446935 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:13.446940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.447761 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.447764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:13.448462 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:13.448471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.449178 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.449182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:13.449261 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:13.453541 (+ 4280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:13.453554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:13.455437 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":135,"threads_started":1}
W20260620 12:20:13.455724 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:13.446034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48185 (local address 127.0.98.62:45433)
0620 12:20:13.446189 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:13.446195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:13.446370 (+ 175us) server_negotiation.cc:407] Connection header received
0620 12:20:13.446470 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:13.446472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:13.446518 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:13.446594 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:13.447040 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.447565 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.448681 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.449013 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.449173 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:13.455573 (+ 6400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48185: BlockingRecv error: recv got EOF from 127.0.98.2:48185 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 12:20:13 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:13 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:13.649371 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:13.642870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:46321)
0620 12:20:13.643218 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:13.643232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:13.643394 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:13.643821 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:13.643824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:13.644012 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:13.644223 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:13.644234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.645469 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.645474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:13.646124 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:13.646134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.646624 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.646627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:13.646705 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:13.647303 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:13.647325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:13.649246 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":85,"threads_started":1}
W20260620 12:20:13.649679 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:13.642983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46321 (local address 127.0.98.62:45433)
0620 12:20:13.643166 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:13.643171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:13.643308 (+ 137us) server_negotiation.cc:407] Connection header received
0620 12:20:13.643554 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:13.643560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:13.643627 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:13.643755 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:13.644383 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.645348 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.646248 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:13.646482 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:13.646571 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:13.649388 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46321: BlockingRecv error: recv got EOF from 127.0.98.3:46321 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 12:20:14 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:14 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:14.279817 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:14.274348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:59927)
0620 12:20:14.274637 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:14.274651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:14.274748 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:14.274991 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:14.274994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:14.275171 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:14.275272 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:14.275281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.275980 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.275983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:14.276548 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:14.276596 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.277033 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.277036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:14.277180 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:14.277660 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:14.277671 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:14.279557 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260620 12:20:14.279862 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:14.274424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59927 (local address 127.0.98.62:45433)
0620 12:20:14.274661 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:14.274664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:14.274700 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:20:14.274834 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:14.274836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:14.274879 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:14.274954 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:14.275373 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.275875 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.276695 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.276910 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.276988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:14.279699 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59927: BlockingRecv error: recv got EOF from 127.0.98.1:59927 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
Jun 20 12:20:14 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:14 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:14.461745 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:14.456324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:54305)
0620 12:20:14.456573 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:14.456589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:14.456725 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:14.456996 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:14.456998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:14.457237 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:14.457330 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:14.457335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.458084 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.458089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:14.458640 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:14.458649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.459121 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.459124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:14.459210 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:14.459759 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:14.459770 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:14.461574 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":66,"threads_started":1}
W20260620 12:20:14.461802 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:14.456411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54305 (local address 127.0.98.62:45433)
0620 12:20:14.456567 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:14.456571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:14.456664 (+ 93us) server_negotiation.cc:407] Connection header received
0620 12:20:14.456826 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:14.456831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:14.456880 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:14.456973 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:14.457442 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.457973 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.458778 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.459009 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.459121 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:14.461685 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54305: BlockingRecv error: recv got EOF from 127.0.98.2:54305 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 12:20:14 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:14 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:14.655898 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:14.650201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:50891)
0620 12:20:14.650555 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:14.650570 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:14.650708 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:14.650970 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:14.650973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:14.651196 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:14.651284 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:14.651289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.652054 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.652060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:14.652623 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:14.652631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.653124 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.653128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:14.653220 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:14.653745 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:14.653756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:14.655766 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":133,"threads_started":1}
W20260620 12:20:14.656054 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:14.650374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50891 (local address 127.0.98.62:45433)
0620 12:20:14.650526 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:14.650531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:14.650648 (+ 117us) server_negotiation.cc:407] Connection header received
0620 12:20:14.650807 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:14.650811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:14.650861 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:14.650952 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:14.651405 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.651945 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.652752 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:14.652987 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:14.653124 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:14.655913 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50891: BlockingRecv error: recv got EOF from 127.0.98.3:50891 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 12:20:15 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:15 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:15.286491 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:15.280924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:40225)
0620 12:20:15.281230 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:15.281244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:15.281334 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:15.281592 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:15.281595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:15.281862 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:15.281991 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:15.282000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.282944 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.282947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:15.283495 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:15.283502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.283896 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.283898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:15.283957 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:15.284436 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:15.284446 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:15.286301 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
W20260620 12:20:15.286540 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:15.280979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40225 (local address 127.0.98.62:45433)
0620 12:20:15.281268 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:15.281271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:15.281288 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:20:15.281419 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:15.281422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:15.281464 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:15.281568 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:15.282152 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.282846 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.283593 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.283803 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.283876 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:15.286420 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40225: BlockingRecv error: recv got EOF from 127.0.98.1:40225 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":123,"threads_started":1}
Jun 20 12:20:15 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:15 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:15.468128 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:15.462510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:39593)
0620 12:20:15.462779 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:15.462799 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:15.462895 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:15.463132 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:15.463135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:15.463306 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:15.463395 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:15.463401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.464114 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.464116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:15.464659 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:15.464666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.465169 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.465171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:15.465253 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:15.465770 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:15.465782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:15.467980 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
W20260620 12:20:15.468246 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:15.462639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39593 (local address 127.0.98.62:45433)
0620 12:20:15.462818 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:15.462821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:15.462851 (+ 30us) server_negotiation.cc:407] Connection header received
0620 12:20:15.462982 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:15.462984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:15.463029 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:15.463099 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:15.463498 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.464023 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.464796 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.465024 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.465143 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:15.468121 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39593: BlockingRecv error: recv got EOF from 127.0.98.2:39593 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 12:20:15 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:15 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:15.662444 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:15.656772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:56717)
0620 12:20:15.657155 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:15.657169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:15.657304 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:15.657604 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:15.657607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:15.657809 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:15.657897 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:15.657902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.658680 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.658685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:15.659260 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:15.659268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.659737 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.659741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:15.659826 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:15.660317 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:15.660330 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:15.662278 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
W20260620 12:20:15.662513 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:15.656881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56717 (local address 127.0.98.62:45433)
0620 12:20:15.657108 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:15.657113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:15.657245 (+ 132us) server_negotiation.cc:407] Connection header received
0620 12:20:15.657405 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:15.657409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:15.657474 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:15.657581 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:15.658028 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.658565 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.659386 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:15.659616 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:15.659730 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:15.662406 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56717: BlockingRecv error: recv got EOF from 127.0.98.3:56717 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 20 12:20:16 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:16 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:16.292659 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:16.287321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:54281)
0620 12:20:16.287570 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:16.287586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:16.287676 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:16.287885 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:16.287888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:16.288056 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:16.288160 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:16.288165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.288937 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.288939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:16.289518 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:16.289526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.289962 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.289964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:16.290027 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:16.290530 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:16.290541 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:16.292480 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
W20260620 12:20:16.292735 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:16.287401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54281 (local address 127.0.98.62:45433)
0620 12:20:16.287667 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:16.287670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:16.287684 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:16.287748 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:16.287750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:16.287791 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:16.287859 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:16.288269 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.288836 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.289627 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.289850 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.289942 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:16.292609 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54281: BlockingRecv error: recv got EOF from 127.0.98.1:54281 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
Jun 20 12:20:16 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:16 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:16.475068 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:16.469089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:44617)
0620 12:20:16.469386 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:16.469400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:16.469502 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:16.469754 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:16.469756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:16.469928 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:16.470019 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:16.470024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.470874 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.470877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:16.471532 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:16.471541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.472175 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.472179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:16.472269 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:16.472860 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:16.472879 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:16.474938 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
W20260620 12:20:16.475232 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:16.469190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44617 (local address 127.0.98.62:45433)
0620 12:20:16.469373 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:16.469378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:16.469453 (+ 75us) server_negotiation.cc:407] Connection header received
0620 12:20:16.469577 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:16.469579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:16.469620 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:16.469691 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:16.470139 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.470775 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.471656 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.472038 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.472200 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:16.475076 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44617: BlockingRecv error: recv got EOF from 127.0.98.2:44617 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 12:20:16 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:16 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:16.669232 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:16.663251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:56273)
0620 12:20:16.663644 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:16.663659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:16.663762 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:16.664016 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:16.664019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:16.664207 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:16.664317 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:16.664323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.665197 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.665204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:16.665961 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:16.665971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.666469 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.666472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:16.666540 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:16.667079 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:16.667091 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:16.669121 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
W20260620 12:20:16.669641 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:16.663419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56273 (local address 127.0.98.62:45433)
0620 12:20:16.663583 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:16.663587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:16.663740 (+ 153us) server_negotiation.cc:407] Connection header received
0620 12:20:16.663828 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:16.663830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:16.663874 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:16.663943 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:16.664431 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.664977 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.666084 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:16.666294 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:16.666376 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:16.669543 (+ 3167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56273: BlockingRecv error: recv got EOF from 127.0.98.3:56273 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 12:20:17 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:17 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:17.299500 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:17.293597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:52181)
0620 12:20:17.293901 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:17.293915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:17.294001 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:17.294192 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:17.294194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:17.294389 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:17.294485 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:17.294490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.295271 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.295275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:17.295932 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:17.295942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.296413 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.296415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:17.296475 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:17.297095 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:17.297106 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:17.299342 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
W20260620 12:20:17.299583 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:17.293759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52181 (local address 127.0.98.62:45433)
0620 12:20:17.294011 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:17.294014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:17.294027 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:20:17.294063 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:17.294065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:17.294101 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:17.294185 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:17.294592 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.295146 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.296086 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.296307 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.296398 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:17.299481 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52181: BlockingRecv error: recv got EOF from 127.0.98.1:52181 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":110,"threads_started":1}
Jun 20 12:20:17 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:17 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:17.481320 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:17.475906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:44165)
0620 12:20:17.476248 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:17.476263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:17.476355 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:17.476573 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:17.476575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:17.476751 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:17.476846 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:17.476851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.477682 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.477685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:17.478233 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:17.478241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.478740 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.478743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:17.478807 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:17.479319 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:17.479330 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:17.481188 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":121,"threads_started":1}
W20260620 12:20:17.481415 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:17.476045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44165 (local address 127.0.98.62:45433)
0620 12:20:17.476244 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:17.476248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:17.476311 (+ 63us) server_negotiation.cc:407] Connection header received
0620 12:20:17.476419 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:17.476421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:17.476465 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:17.476530 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:17.476948 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.477577 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.478341 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.478561 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.478637 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:17.481323 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44165: BlockingRecv error: recv got EOF from 127.0.98.2:44165 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 12:20:17 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:17 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:17.675307 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:17.670011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:32989)
0620 12:20:17.670310 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:17.670325 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:17.670475 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:17.670759 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:17.670761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:17.670927 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:17.671037 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:17.671042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.671827 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.671830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:17.672398 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:17.672406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.672808 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.672811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:17.672883 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:17.673363 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:17.673375 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:17.675178 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
W20260620 12:20:17.675397 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:17.670059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32989 (local address 127.0.98.62:45433)
0620 12:20:17.670191 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:17.670196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:17.670438 (+ 242us) server_negotiation.cc:407] Connection header received
0620 12:20:17.670615 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:17.670617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:17.670661 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:17.670727 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:17.671131 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.671736 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.672493 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:17.672698 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:17.672775 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:17.675308 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32989: BlockingRecv error: recv got EOF from 127.0.98.3:32989 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 12:20:18 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:18 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:18.306139 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:18.300300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:54931)
0620 12:20:18.300566 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:18.300586 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:18.300676 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:18.300987 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:18.300990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:18.301242 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:18.301346 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:18.301352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.302141 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.302143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:18.302704 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:18.302711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.303142 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.303144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:18.303201 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:18.303751 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:18.303763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:18.305968 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
W20260620 12:20:18.306439 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:18.300433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54931 (local address 127.0.98.62:45433)
0620 12:20:18.300683 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:18.300687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:18.300710 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:20:18.300745 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:18.300747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:18.300785 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:18.300867 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:18.301505 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.302037 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.302820 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.303040 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.303121 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:18.306129 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54931: BlockingRecv error: recv got EOF from 127.0.98.1:54931 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
Jun 20 12:20:18 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:18 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 12:20:18.492306 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:18.482250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54645 (local address 127.0.98.62:45433)
0620 12:20:18.482454 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:18.482458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:18.482486 (+ 28us) server_negotiation.cc:407] Connection header received
0620 12:20:18.482618 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:18.482620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:18.482660 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:18.482726 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:18.483128 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.483651 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.484442 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.484645 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.484717 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:18.492212 (+ 7495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54645: BlockingRecv error: recv got EOF from 127.0.98.2:54645 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 12:20:18.492460 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:18.482130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:54645)
0620 12:20:18.482421 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:18.482436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:18.482531 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:18.482770 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:18.482772 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:18.482942 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:18.483031 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:18.483037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.483752 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.483755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:18.484337 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:18.484346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.484740 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.484743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:18.484801 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:18.490124 (+ 5323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:18.490137 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:18.492115 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":65,"threads_started":1}
Jun 20 12:20:18 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:18 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:18.681622 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:18.676129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:52143)
0620 12:20:18.676384 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:18.676403 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:18.676502 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:18.676749 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:18.676751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:18.676936 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:18.677029 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:18.677034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.677764 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.677766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:18.678423 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:18.678432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.679013 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.679019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:18.679101 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:18.679648 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:18.679659 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:18.681511 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
W20260620 12:20:18.681797 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:18.676188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52143 (local address 127.0.98.62:45433)
0620 12:20:18.676335 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:18.676339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:18.676479 (+ 140us) server_negotiation.cc:407] Connection header received
0620 12:20:18.676592 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:18.676594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:18.676639 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:18.676714 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:18.677158 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.677670 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.678541 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:18.678848 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:18.678982 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:18.681652 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52143: BlockingRecv error: recv got EOF from 127.0.98.3:52143 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 12:20:19 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:19 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:19.312497 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:19.306981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:43369)
0620 12:20:19.307294 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:19.307323 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:19.307420 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:19.307707 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:19.307709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:19.307882 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:19.307975 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:19.307980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.308830 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.308833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:19.309473 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:19.309482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.309922 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.309925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:19.309984 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:19.310563 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:19.310575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:19.312387 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
W20260620 12:20:19.312652 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:19.307050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43369 (local address 127.0.98.62:45433)
0620 12:20:19.307312 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:19.307316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:19.307373 (+ 57us) server_negotiation.cc:407] Connection header received
0620 12:20:19.307538 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:19.307540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:19.307582 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:19.307659 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:19.308079 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.308690 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.309584 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.309800 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.309901 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:19.312513 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43369: BlockingRecv error: recv got EOF from 127.0.98.1:43369 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":66,"threads_started":1}
Jun 20 12:20:19 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:19 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:19.499233 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:19.493213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:59195)
0620 12:20:19.493505 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:19.493519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:19.493612 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:19.493853 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:19.493855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:19.494047 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:19.494140 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:19.494146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.495022 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.495025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:19.495622 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:19.495631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.496137 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.496140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:19.496273 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:19.496873 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:19.496885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:19.499116 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
W20260620 12:20:19.499461 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:19.493345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59195 (local address 127.0.98.62:45433)
0620 12:20:19.493497 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:19.493501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:19.493566 (+ 65us) server_negotiation.cc:407] Connection header received
0620 12:20:19.493691 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:19.493693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:19.493734 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:19.493805 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:19.494231 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.494914 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.495756 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.495986 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.496077 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:19.499343 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59195: BlockingRecv error: recv got EOF from 127.0.98.2:59195 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 12:20:19 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:19 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:19.688583 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:19.682387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:55939)
0620 12:20:19.682689 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:19.682703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:19.682802 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:19.683044 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:19.683046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:19.683223 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:19.683322 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:19.683328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.684047 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.684049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:19.684631 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:19.684640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.685421 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.685424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:19.685495 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:19.686014 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:19.686031 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:19.688419 (+ 2388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
W20260620 12:20:19.688696 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:19.682447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55939 (local address 127.0.98.62:45433)
0620 12:20:19.682594 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:19.682599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:19.682781 (+ 182us) server_negotiation.cc:407] Connection header received
0620 12:20:19.682897 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:19.682899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:19.682943 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:19.683010 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:19.683421 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.683936 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.684767 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:19.684987 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:19.685405 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:19.688574 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55939: BlockingRecv error: recv got EOF from 127.0.98.3:55939 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 12:20:20 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:20 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:20.318692 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:20.313363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:59509)
0620 12:20:20.313658 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:20.313672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:20.313760 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:20.313979 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:20.313981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:20.314147 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:20.314248 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:20.314253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.314961 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.314963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:20.315525 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:20.315532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.315941 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.315944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:20.316000 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:20.316465 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:20.316476 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:20.318571 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
W20260620 12:20:20.318902 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:20.313488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59509 (local address 127.0.98.62:45433)
0620 12:20:20.313697 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:20.313701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:20.313713 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:20:20.313848 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:20.313850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:20.313893 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:20.313967 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:20.314342 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.314869 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.315626 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.315846 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.315920 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:20.318721 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59509: BlockingRecv error: recv got EOF from 127.0.98.1:59509 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
Jun 20 12:20:20 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:20 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:20.505352 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:20.499959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:35897)
0620 12:20:20.500237 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:20.500251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:20.500385 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:20.500656 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:20.500659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:20.500879 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:20.500968 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:20.500973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.501899 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.501905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:20.502477 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:20.502486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.502923 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.502927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:20.503012 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:20.503478 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:20.503490 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:20.505182 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260620 12:20:20.505407 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:20.500011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35897 (local address 127.0.98.62:45433)
0620 12:20:20.500179 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:20.500184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:20.500325 (+ 141us) server_negotiation.cc:407] Connection header received
0620 12:20:20.500491 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:20.500495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:20.500549 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:20.500638 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:20.501113 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.501793 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.502596 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.502817 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.502921 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:20.505303 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35897: BlockingRecv error: recv got EOF from 127.0.98.2:35897 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 12:20:20 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:20 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:20.695479 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:20.689307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:43021)
0620 12:20:20.689575 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:20.689588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:20.689682 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:20.689907 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:20.689909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:20.690079 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:20.690250 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:20.690258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.691261 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.691265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:20.692161 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:20.692172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.692710 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.692713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:20.692774 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:20.693480 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:20.693502 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:20.695311 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
W20260620 12:20:20.695521 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:20.689417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43021 (local address 127.0.98.62:45433)
0620 12:20:20.689557 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:20.689561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:20.689650 (+ 89us) server_negotiation.cc:407] Connection header received
0620 12:20:20.689762 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:20.689764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:20.689809 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:20.689881 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:20.690367 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.691140 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.692277 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:20.692602 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:20.692721 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:20.695422 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43021: BlockingRecv error: recv got EOF from 127.0.98.3:43021 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 12:20:21 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:21 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:21.325388 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:21.319586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:50469)
0620 12:20:21.319949 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:21.319966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:21.320080 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:21.320347 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:21.320351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:21.320583 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:21.320696 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:21.320702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.321559 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.321564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:21.322148 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:21.322156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.322607 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.322610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:21.322695 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:21.323259 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:21.323271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:21.325221 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
W20260620 12:20:21.325464 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:21.319657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50469 (local address 127.0.98.62:45433)
0620 12:20:21.319976 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:21.320038 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:21.320053 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:20:21.320171 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:21.320176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:21.320233 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:21.320327 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:21.320824 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.321450 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.322272 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.322501 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.322603 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:21.325339 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50469: BlockingRecv error: recv got EOF from 127.0.98.1:50469 (error 108)
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":68,"threads_started":1}
Jun 20 12:20:21 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:21 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:21.511490 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:21.506163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:57439)
0620 12:20:21.506426 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:21.506439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:21.506536 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:21.506812 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:21.506815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:21.506991 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:21.507081 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:21.507086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.507878 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.507885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:21.508444 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:21.508494 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.508930 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.508934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:21.509016 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:21.509568 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:21.509579 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:21.511379 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
W20260620 12:20:21.511662 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:21.506233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57439 (local address 127.0.98.62:45433)
0620 12:20:21.506362 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:21.506366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:21.506502 (+ 136us) server_negotiation.cc:407] Connection header received
0620 12:20:21.506618 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:21.506620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:21.506670 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:21.506744 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:21.507181 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.507753 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.508595 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.508820 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.508925 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:21.511503 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57439: BlockingRecv error: recv got EOF from 127.0.98.2:57439 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 12:20:21 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:21 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:21.702811 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:21.696490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:43885)
0620 12:20:21.697266 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:21.697281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:21.697402 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:21.697797 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:21.697799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:21.698094 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:21.698211 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:21.698218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.699064 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.699067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:21.699635 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:21.699644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.700112 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.700115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:21.700190 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:21.700701 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:21.700714 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:21.702698 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":111,"threads_started":1}
W20260620 12:20:21.702930 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:21.696575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43885 (local address 127.0.98.62:45433)
0620 12:20:21.696696 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:21.696701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:21.697364 (+ 663us) server_negotiation.cc:407] Connection header received
0620 12:20:21.697478 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:21.697481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:21.697537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:21.697631 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:21.698353 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.698958 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.699732 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:21.699981 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:21.700076 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:21.702819 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43885: BlockingRecv error: recv got EOF from 127.0.98.3:43885 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 12:20:22 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:22 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:22.331931 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:22.326216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:34085)
0620 12:20:22.326470 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:22.326483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:22.326569 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:22.326803 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:22.326805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:22.326980 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:22.327100 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:22.327106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.327834 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.327836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:22.328508 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:22.328517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.329121 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.329124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:22.329222 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:22.330026 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:22.330043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:22.331824 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
W20260620 12:20:22.332218 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:22.326277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34085 (local address 127.0.98.62:45433)
0620 12:20:22.326518 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:22.326521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:22.326534 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:20:22.326650 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:22.326652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:22.326694 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:22.326770 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:22.327204 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.327737 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.328615 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.328950 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.329099 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:22.332070 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34085: BlockingRecv error: recv got EOF from 127.0.98.1:34085 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":59,"threads_started":1}
Jun 20 12:20:22 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:22 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:22.518435 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:22.512322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:43849)
0620 12:20:22.512616 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:22.512639 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:22.512783 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:22.513293 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:22.513296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:22.513528 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:22.513626 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:22.513632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.514413 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.514419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:22.515010 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:22.515018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.515497 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.515501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:22.515587 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:22.516212 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:22.516224 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:22.518263 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
W20260620 12:20:22.518493 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:22.512359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43849 (local address 127.0.98.62:45433)
0620 12:20:22.512506 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:22.512511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:22.512731 (+ 220us) server_negotiation.cc:407] Connection header received
0620 12:20:22.513107 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:22.513110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:22.513169 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:22.513265 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:22.513750 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.514289 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.515147 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.515383 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.515498 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:22.518388 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43849: BlockingRecv error: recv got EOF from 127.0.98.2:43849 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 12:20:22 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:22 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:22.709364 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:22.703735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:40175)
0620 12:20:22.704061 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:22.704077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:22.704196 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:22.704448 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:22.704451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:22.704654 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:22.704740 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:22.704745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.705583 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.705587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:22.706156 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:22.706164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.706614 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.706618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:22.706699 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:22.707217 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:22.707232 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:22.709213 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
W20260620 12:20:22.709556 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:22.703863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40175 (local address 127.0.98.62:45433)
0620 12:20:22.704009 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:22.704014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:22.704147 (+ 133us) server_negotiation.cc:407] Connection header received
0620 12:20:22.704296 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:22.704299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:22.704353 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:22.704432 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:22.704847 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.705446 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.706279 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:22.706503 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:22.706612 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:22.709403 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40175: BlockingRecv error: recv got EOF from 127.0.98.3:40175 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 12:20:23 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:23 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:23.338197 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:23.332680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:35877)
0620 12:20:23.332933 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:23.332949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:23.333093 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:23.333358 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:23.333361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:23.333539 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:23.333634 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:23.333639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.334531 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.334533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:23.335086 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:23.335094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.335538 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.335540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:23.335597 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:23.336142 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:23.336156 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:23.338057 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
W20260620 12:20:23.338403 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:23.332841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35877 (local address 127.0.98.62:45433)
0620 12:20:23.333126 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:23.333131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:23.333145 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:23.333181 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:23.333183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:23.333226 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:23.333342 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:23.333737 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.334424 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.335185 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.335439 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.335525 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:23.338200 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35877: BlockingRecv error: recv got EOF from 127.0.98.1:35877 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
Jun 20 12:20:23 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:23 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:23.528671 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:23.519241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:58927)
0620 12:20:23.519563 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:23.519583 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:23.519736 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:23.519996 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:23.520000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:23.520208 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:23.520296 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:23.520302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.521118 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.521123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:23.521675 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:23.521683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.522125 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.522129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:23.522214 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:23.526389 (+ 4175us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:23.526404 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:23.528480 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
W20260620 12:20:23.528800 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:23.519366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58927 (local address 127.0.98.62:45433)
0620 12:20:23.519532 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:23.519537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:23.519675 (+ 138us) server_negotiation.cc:407] Connection header received
0620 12:20:23.519840 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:23.519843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:23.519893 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:23.519977 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:23.520423 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.520980 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.521789 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.522017 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.522123 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:23.528646 (+ 6523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58927: BlockingRecv error: recv got EOF from 127.0.98.2:58927 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 12:20:23 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:23 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:23.715503 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:23.710260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:33095)
0620 12:20:23.710514 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:23.710527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:23.710631 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:23.710841 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:23.710844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:23.711034 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:23.711124 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:23.711129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.711970 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.711975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:23.712598 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:23.712607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.713084 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.713088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:23.713172 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:23.713692 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:23.713704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:23.715393 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
W20260620 12:20:23.715680 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:23.710304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33095 (local address 127.0.98.62:45433)
0620 12:20:23.710434 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:23.710439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:23.710594 (+ 155us) server_negotiation.cc:407] Connection header received
0620 12:20:23.710698 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:23.710700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:23.710746 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:23.710812 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:23.711250 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.711861 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.712716 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:23.712940 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:23.713046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:23.715524 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33095: BlockingRecv error: recv got EOF from 127.0.98.3:33095 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 12:20:24 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:24 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:24.344074 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:24.339002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:51509)
0620 12:20:24.339252 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:24.339285 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:24.339405 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:24.339634 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:24.339636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:24.339809 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:24.339900 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:24.339904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.340593 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.340595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:24.341159 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:24.341167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.341559 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.341561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:24.341621 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:24.342088 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:24.342099 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:24.343902 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
W20260620 12:20:24.344133 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:24.339079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51509 (local address 127.0.98.62:45433)
0620 12:20:24.339312 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:24.339317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:24.339336 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:20:24.339499 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:24.339502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:24.339546 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:24.339622 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:24.339990 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.340504 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.341248 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.341465 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.341540 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:24.344016 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51509: BlockingRecv error: recv got EOF from 127.0.98.1:51509 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
Jun 20 12:20:24 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:24 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:24.535161 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:24.529477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:41349)
0620 12:20:24.529818 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:24.529834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:24.529953 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:24.530206 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:24.530208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:24.530423 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:24.530512 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:24.530517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.531255 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.531261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:24.531832 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:24.531840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.532285 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.532289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:24.532382 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:24.532966 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:24.532983 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:24.534997 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":87,"threads_started":1}
W20260620 12:20:24.535209 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:24.529600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41349 (local address 127.0.98.62:45433)
0620 12:20:24.529783 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:24.529788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:24.529920 (+ 132us) server_negotiation.cc:407] Connection header received
0620 12:20:24.530042 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:24.530047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:24.530095 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:24.530181 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:24.530629 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.531148 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.531950 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.532170 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.532271 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:24.535109 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41349: BlockingRecv error: recv got EOF from 127.0.98.2:41349 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 12:20:24 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:24 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:24.721504 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:24.716223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:53557)
0620 12:20:24.716479 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:24.716498 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:24.716623 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:24.716898 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:24.716901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:24.717143 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:24.717237 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:24.717242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.717974 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.717980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:24.718533 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:24.718540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.718979 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.718982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:24.719064 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:24.719546 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:24.719557 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:24.721337 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
W20260620 12:20:24.721565 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:24.716294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53557 (local address 127.0.98.62:45433)
0620 12:20:24.716455 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:24.716460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:24.716563 (+ 103us) server_negotiation.cc:407] Connection header received
0620 12:20:24.716715 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:24.716718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:24.716770 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:24.716852 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:24.717343 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.717872 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.718642 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:24.718874 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:24.718977 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:24.721456 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53557: BlockingRecv error: recv got EOF from 127.0.98.3:53557 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 12:20:25 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:25 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:25.350200 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:25.345013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:47993)
0620 12:20:25.345318 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:25.345340 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:25.345433 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:25.345676 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:25.345678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:25.345853 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:25.345946 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:25.345950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.346722 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.346725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:25.347307 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:25.347315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.347724 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.347734 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:25.347791 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:25.348271 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:25.348282 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:25.350088 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":72,"threads_started":1}
W20260620 12:20:25.350441 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:25.345133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47993 (local address 127.0.98.62:45433)
0620 12:20:25.345413 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:25.345416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:25.345432 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:20:25.345512 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:25.345515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:25.345559 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:25.345646 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:25.346067 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.346618 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.347412 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.347630 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.347709 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:25.350263 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47993: BlockingRecv error: recv got EOF from 127.0.98.1:47993 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
Jun 20 12:20:25 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:25 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:25.540977 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:25.535878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:47833)
0620 12:20:25.536144 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:25.536157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:25.536240 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:25.536453 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:25.536455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:25.536625 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:25.536717 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:25.536723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.537517 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.537519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:25.538084 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:25.538092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.538524 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.538527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:25.538587 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:25.539058 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:25.539068 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:25.540869 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
W20260620 12:20:25.541144 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:25.536073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47833 (local address 127.0.98.62:45433)
0620 12:20:25.536229 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:25.536234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:25.536248 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:20:25.536309 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:25.536312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:25.536354 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:25.536433 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:25.536824 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.537423 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.538199 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.538421 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.538507 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:25.540993 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47833: BlockingRecv error: recv got EOF from 127.0.98.2:47833 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 12:20:25 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:25 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:25.727977 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:25.722358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:43257)
0620 12:20:25.722656 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:25.722670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:25.722775 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:25.723038 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:25.723041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:25.723227 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:25.723327 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:25.723332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.724194 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.724198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:25.724879 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:25.724889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.725374 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.725377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:25.725446 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:25.725984 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:25.725997 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:25.727851 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
W20260620 12:20:25.728184 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:25.722418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43257 (local address 127.0.98.62:45433)
0620 12:20:25.722538 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:25.722542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:25.722727 (+ 185us) server_negotiation.cc:407] Connection header received
0620 12:20:25.722861 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:25.722863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:25.722907 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:25.722966 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:25.723471 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.724068 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.725005 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:25.725273 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:25.725355 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:25.728013 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43257: BlockingRecv error: recv got EOF from 127.0.98.3:43257 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 12:20:26 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:26 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:26.355988 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:26.351063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:53019)
0620 12:20:26.351329 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:26.351342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:26.351428 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:26.351651 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:26.351653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:26.351831 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:26.351923 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:26.351928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.352625 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.352628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:26.353260 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:26.353269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.353667 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.353670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:26.353726 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:26.354179 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:26.354207 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:26.355868 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260620 12:20:26.356118 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:26.351135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53019 (local address 127.0.98.62:45433)
0620 12:20:26.351420 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:26.351424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:26.351436 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:20:26.351506 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:26.351508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:26.351548 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:26.351616 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:26.352020 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.352535 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.353357 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.353572 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.353646 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:26.355993 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53019: BlockingRecv error: recv got EOF from 127.0.98.1:53019 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
Jun 20 12:20:26 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:26 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:26.547565 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:26.541870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:34103)
0620 12:20:26.542149 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:26.542169 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:26.542267 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:26.542525 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:26.542528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:26.542743 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:26.542838 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:26.542843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.543626 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.543631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:26.544364 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:26.544375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.544976 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.544980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:26.545114 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:26.545639 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:26.545651 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:26.547440 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
W20260620 12:20:26.547706 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:26.541938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34103 (local address 127.0.98.62:45433)
0620 12:20:26.542106 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:26.542110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:26.542227 (+ 117us) server_negotiation.cc:407] Connection header received
0620 12:20:26.542350 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:26.542352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:26.542393 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:26.542456 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:26.542944 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.543464 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.544514 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.544853 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.544957 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:26.547583 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34103: BlockingRecv error: recv got EOF from 127.0.98.2:34103 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 12:20:26 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:26 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:26.734042 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:26.728771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.3:46411)
0620 12:20:26.729028 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:26.729081 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:26.729202 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:26.729417 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:26.729419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:26.729590 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:26.729688 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:26.729693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.730461 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.730464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:26.731012 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:26.731020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.731459 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.731461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:26.731522 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:26.732008 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:26.732020 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:26.733918 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
W20260620 12:20:26.734162 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:26.728816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46411 (local address 127.0.98.62:45433)
0620 12:20:26.728935 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:26.728938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:26.729143 (+ 205us) server_negotiation.cc:407] Connection header received
0620 12:20:26.729269 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:26.729271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:26.729317 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:26.729384 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:26.729820 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.730359 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.731142 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:26.731356 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:26.731447 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:26.734048 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46411: BlockingRecv error: recv got EOF from 127.0.98.3:46411 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 12:20:27 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:27 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:27.362783 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:27.356995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.1:40845)
0620 12:20:27.357282 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:27.357296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:27.357385 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:27.357623 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:27.357625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:27.357798 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:27.357889 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:27.357894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.358891 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.358894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:27.359572 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:27.359587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.360046 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.360050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:27.360113 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:27.360728 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:27.360742 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:27.362661 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":123,"threads_started":1}
W20260620 12:20:27.362967 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:27.357068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40845 (local address 127.0.98.62:45433)
0620 12:20:27.357307 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:27.357310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:27.357348 (+ 38us) server_negotiation.cc:407] Connection header received
0620 12:20:27.357469 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:27.357471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:27.357511 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:27.357589 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:27.357987 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.358753 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.359702 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.359922 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.360005 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:27.362793 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40845: BlockingRecv error: recv got EOF from 127.0.98.1:40845 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":55,"threads_started":1}
Jun 20 12:20:27 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 12:20:27 dist-test-slave-9b3m krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 12:20:27.554312 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:27.548543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.98.2:57109)
0620 12:20:27.548795 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:27.548810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:27.548922 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:27.549225 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:27.549228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:27.549431 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:27.549535 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:27.549541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.550259 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.550262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:27.550933 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:27.550942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.551560 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.551562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:27.551658 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:27.552332 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:27.552344 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:27.554199 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45433: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
W20260620 12:20:27.554566 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:27.548628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57109 (local address 127.0.98.62:45433)
0620 12:20:27.548764 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:27.548768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:27.548884 (+ 116us) server_negotiation.cc:407] Connection header received
0620 12:20:27.548998 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:27.549000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:27.549039 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:27.549148 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:27.549653 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.550156 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.551077 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.551416 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.551540 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:27.554344 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57109: BlockingRecv error: recv got EOF from 127.0.98.2:57109 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:20:27.631512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 9131
I20260620 12:20:27.637177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 9283
I20260620 12:20:27.642359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 9415
I20260620 12:20:27.647732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 9061
2026-06-20T12:20:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17980 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T12:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:20:27Z Disabled control of system clock
I20260620 12:20:27.679901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 12:20:27.787966 9612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:27.788220 9612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:27.788254 9612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:27.792240 9612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:20:27.792290 9612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:27.792305 9612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:27.792321 9612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:20:27.792372 9612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:20:27.796972 9612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:27.797973 9612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:27.799289 9612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:27.807148 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:27.807232 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:27.807349 9612 server_base.cc:1034] running on GCE node
W20260620 12:20:27.807148 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:27.807953 9612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:27.809000 9612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:27.810227 9612 hybrid_clock.cc:648] HybridClock initialized: now 1781958027810212 us; error 28 us; skew 500 ppm
I20260620 12:20:27.811973 9612 webserver.cc:460] Webserver started at http://127.0.98.62:36355/ using document root <none> and password file <none>
I20260620 12:20:27.812559 9612 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:27.812619 9612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:27.812826 9612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:27.814904 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "3918b92a00b8482a8feaa437d8a069d5"
format_stamp: "Formatted at 2026-06-20 12:20:27 on dist-test-slave-9b3m"
server_key: "8f8f45461954f5071f9d73f37464b3a2"
server_key_iv: "4ba63f9efd423b0e9dd2f1ac2c11f2ab"
server_key_version: "encryptionkey@0"
I20260620 12:20:27.815442 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "3918b92a00b8482a8feaa437d8a069d5"
format_stamp: "Formatted at 2026-06-20 12:20:27 on dist-test-slave-9b3m"
server_key: "8f8f45461954f5071f9d73f37464b3a2"
server_key_iv: "4ba63f9efd423b0e9dd2f1ac2c11f2ab"
server_key_version: "encryptionkey@0"
I20260620 12:20:27.818881 9612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:20:27.821110 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:27.822248 9612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:20:27.822381 9612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "3918b92a00b8482a8feaa437d8a069d5"
format_stamp: "Formatted at 2026-06-20 12:20:27 on dist-test-slave-9b3m"
server_key: "8f8f45461954f5071f9d73f37464b3a2"
server_key_iv: "4ba63f9efd423b0e9dd2f1ac2c11f2ab"
server_key_version: "encryptionkey@0"
I20260620 12:20:27.822485 9612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:27.840394 9612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:27.841014 9612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:27.841239 9612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:27.848163 9612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38417
I20260620 12:20:27.848189 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38417 every 8 connection(s)
I20260620 12:20:27.849120 9612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:20:27.852167 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:27.855123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 9612
I20260620 12:20:27.855260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:20:27.855531 392 external_mini_cluster.cc:1428] Setting key a5a56f6c337edf2d35b759d95e4e9988
I20260620 12:20:27.857995 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5: Bootstrap starting.
I20260620 12:20:27.860391 9679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:27.861153 9679 log.cc:824] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:27.863385 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5: No bootstrap required, opened a new log
I20260620 12:20:27.865413 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:27.856903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.0.98.62:38417)
0620 12:20:27.857546 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:27.857561 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:27.857604 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:20:27.858371 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:27.858390 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:27.858789 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:27.859115 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:27.859619 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.860697 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.861767 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:27.862437 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:27.862619 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:27.863401 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:27.863408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:27.863426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:20:27.863917 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:27.863984 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:20:27.864009 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:27.864114 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:27.864223 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 12:20:27.864476 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":151,"threads_started":1}
I20260620 12:20:27.867636 9679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3918b92a00b8482a8feaa437d8a069d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260620 12:20:27.867878 9679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:27.867965 9679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3918b92a00b8482a8feaa437d8a069d5, State: Initialized, Role: FOLLOWER
I20260620 12:20:27.868374 9679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3918b92a00b8482a8feaa437d8a069d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260620 12:20:27.868518 9679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:20:27.868599 9679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:20:27.868695 9679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:27.869642 9679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3918b92a00b8482a8feaa437d8a069d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260620 12:20:27.869957 9679 leader_election.cc:302] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3918b92a00b8482a8feaa437d8a069d5; no voters:
I20260620 12:20:27.870246 9679 leader_election.cc:288] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:20:27.870378 9684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:27.870646 9684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3918b92a00b8482a8feaa437d8a069d5, State: Running, Role: LEADER
I20260620 12:20:27.870999 9684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3918b92a00b8482a8feaa437d8a069d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260620 12:20:27.871408 9679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:20:27.872633 9686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3918b92a00b8482a8feaa437d8a069d5. Latest consensus state: current_term: 1 leader_uuid: "3918b92a00b8482a8feaa437d8a069d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3918b92a00b8482a8feaa437d8a069d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260620 12:20:27.872675 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3918b92a00b8482a8feaa437d8a069d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3918b92a00b8482a8feaa437d8a069d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260620 12:20:27.872761 9686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:27.872769 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:27.873147 9693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:20:27.876200 9693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:20:27.882002 9693 catalog_manager.cc:1347] Generated new cluster ID: 20bcea38aa84493daada42cad475ff41
I20260620 12:20:27.882071 9693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:20:27.900875 9693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:20:27.901793 9693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:20:27.908228 9693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3918b92a00b8482a8feaa437d8a069d5: Generated new TSK 0
I20260620 12:20:27.908847 9693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:20:27.912637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--builtin_ntp_servers=127.0.98.20:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 12:20:28.022004 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:28.022240 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:28.022275 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:28.026309 9703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:28.026357 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:28.026453 9703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:20:28.031057 9703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:28.032094 9703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:28.033465 9703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:28.044579 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.044617 9709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.044610 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:28.044740 9703 server_base.cc:1034] running on GCE node
I20260620 12:20:28.045233 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:28.045832 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:28.046990 9703 hybrid_clock.cc:648] HybridClock initialized: now 1781958028046976 us; error 25 us; skew 500 ppm
I20260620 12:20:28.048744 9703 webserver.cc:460] Webserver started at http://127.0.98.1:37755/ using document root <none> and password file <none>
I20260620 12:20:28.049412 9703 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:28.049474 9703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:28.049700 9703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:28.051676 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "01f2555e18b4466d8b5b21e22528041f"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "5d2d9928cf54e5f7a3fb36175faca9b7"
server_key_iv: "bf6aff80f2efe509c5b74fb3b309adcf"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.052235 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01f2555e18b4466d8b5b21e22528041f"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "5d2d9928cf54e5f7a3fb36175faca9b7"
server_key_iv: "bf6aff80f2efe509c5b74fb3b309adcf"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.055881 9703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 12:20:28.058367 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.059566 9703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:28.059700 9703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "01f2555e18b4466d8b5b21e22528041f"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "5d2d9928cf54e5f7a3fb36175faca9b7"
server_key_iv: "bf6aff80f2efe509c5b74fb3b309adcf"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.059813 9703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:28.083089 9703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:28.083760 9703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:28.083959 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:28.084527 9703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:28.085480 9703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:28.085551 9703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.085618 9703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:28.085665 9703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.094619 9703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40689
I20260620 12:20:28.094668 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40689 every 8 connection(s)
I20260620 12:20:28.095525 9703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:20:28.100926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 9703
I20260620 12:20:28.101015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:20:28.101312 392 external_mini_cluster.cc:1428] Setting key 7707b302e57ecfdd89d11c3d7586839d
I20260620 12:20:28.103332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--builtin_ntp_servers=127.0.98.20:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 12:20:28.104918 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.097326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60021 (local address 127.0.98.62:38417)
0620 12:20:28.097454 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:28.097458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:28.098428 (+ 970us) server_negotiation.cc:407] Connection header received
0620 12:20:28.099789 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:28.099792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:28.099847 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:28.099956 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:28.100824 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.101496 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.102861 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.103087 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.103160 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:28.104370 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:28.104373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:28.104375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:20:28.104386 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:28.104404 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:20:28.104411 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:28.104463 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:28.104658 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 12:20:28.104730 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:20:28.105679 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.097668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:60021)
0620 12:20:28.098287 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:28.098324 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:28.099592 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:28.100120 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:28.100127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:28.100204 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:28.100652 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:28.100689 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.101621 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.101624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:28.102735 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:28.102746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.103236 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.103243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:28.103444 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:28.104074 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:20:28.104079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:28.104133 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:20:28.104212 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:28.104488 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:28.104498 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:28.104507 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:20:28.104647 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 12:20:28.104897 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260620 12:20:28.106796 9831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260620 12:20:28.107015 9831 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:28.107477 9831 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260620 12:20:28.108839 9643 ts_manager.cc:194] Registered new tserver with Master: 01f2555e18b4466d8b5b21e22528041f (127.0.98.1:40689)
I20260620 12:20:28.110008 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60021
W20260620 12:20:28.211937 9834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:28.212199 9834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:28.212236 9834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:28.216306 9834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:28.216368 9834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:28.216465 9834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:20:28.221148 9834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:28.222210 9834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:28.223590 9834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:28.233388 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.233461 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:28.233580 9834 server_base.cc:1034] running on GCE node
W20260620 12:20:28.233477 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:28.234203 9834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:28.234817 9834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:28.236019 9834 hybrid_clock.cc:648] HybridClock initialized: now 1781958028236000 us; error 32 us; skew 500 ppm
I20260620 12:20:28.237892 9834 webserver.cc:460] Webserver started at http://127.0.98.2:41423/ using document root <none> and password file <none>
I20260620 12:20:28.238461 9834 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:28.238548 9834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:28.238794 9834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:28.241101 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "dea9ca61d33542aab6cb9f75124edf9a"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "2d30b8bffd5934775a6e7f8783300f45"
server_key_iv: "483acb0ca35a3cbe2b2cacab852ea77b"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.241670 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dea9ca61d33542aab6cb9f75124edf9a"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "2d30b8bffd5934775a6e7f8783300f45"
server_key_iv: "483acb0ca35a3cbe2b2cacab852ea77b"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.245466 9834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:20:28.248023 9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.249344 9834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:20:28.249471 9834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "dea9ca61d33542aab6cb9f75124edf9a"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "2d30b8bffd5934775a6e7f8783300f45"
server_key_iv: "483acb0ca35a3cbe2b2cacab852ea77b"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.249588 9834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:28.268229 9834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:28.268890 9834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:28.269166 9834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:28.269783 9834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:28.270709 9834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:28.270777 9834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.270840 9834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:28.270912 9834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.279538 9834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38597
I20260620 12:20:28.279548 9961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38597 every 8 connection(s)
I20260620 12:20:28.280474 9834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:20:28.289808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 9834
I20260620 12:20:28.289935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:20:28.289937 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.282276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46167 (local address 127.0.98.62:38417)
0620 12:20:28.282406 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:28.282410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:28.283426 (+ 1016us) server_negotiation.cc:407] Connection header received
0620 12:20:28.285305 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:28.285309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:28.285368 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:28.285472 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:28.286347 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.286877 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.288007 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.288230 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.288340 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:28.289444 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:28.289448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:28.289451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:20:28.289470 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:28.289489 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:20:28.289497 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:28.289584 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:28.289738 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 12:20:28.289797 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:20:28.290249 392 external_mini_cluster.cc:1428] Setting key 071a9295d7731e5d704455ada91a256f
I20260620 12:20:28.290830 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.282656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:46167)
0620 12:20:28.283275 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:28.283313 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:28.285097 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:28.285651 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:28.285658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:28.285740 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:28.286202 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:28.286214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.287006 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.287010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:28.287870 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:28.287878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.288379 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.288383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:28.288547 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:28.289190 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:20:28.289191 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:28.289229 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:20:28.289289 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:28.289584 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:28.289595 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:28.289602 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:20:28.289737 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 12:20:28.290001 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260620 12:20:28.292271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--builtin_ntp_servers=127.0.98.20:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 12:20:28.292332 9962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260620 12:20:28.292557 9962 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:28.293147 9962 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260620 12:20:28.294247 9643 ts_manager.cc:194] Registered new tserver with Master: dea9ca61d33542aab6cb9f75124edf9a (127.0.98.2:38597)
I20260620 12:20:28.295135 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46167
W20260620 12:20:28.403334 9965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:28.403585 9965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:28.403621 9965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:28.407842 9965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:28.407893 9965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:28.407972 9965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:20:28.412734 9965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:28.413786 9965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:28.415079 9965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:28.424479 9973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.424479 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.424503 9970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:28.424682 9965 server_base.cc:1034] running on GCE node
I20260620 12:20:28.425251 9965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:28.425936 9965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:28.427093 9965 hybrid_clock.cc:648] HybridClock initialized: now 1781958028427054 us; error 55 us; skew 500 ppm
I20260620 12:20:28.429011 9965 webserver.cc:460] Webserver started at http://127.0.98.3:34019/ using document root <none> and password file <none>
I20260620 12:20:28.429668 9965 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:28.429728 9965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:28.429931 9965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:28.431945 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "4c259ef22d0d46da9fbe9cdcabe63f2a"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "7e2b3f2f712c674ca94e679ee288d755"
server_key_iv: "1aeecbd942ba1086543068715d0b1512"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.432503 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4c259ef22d0d46da9fbe9cdcabe63f2a"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "7e2b3f2f712c674ca94e679ee288d755"
server_key_iv: "1aeecbd942ba1086543068715d0b1512"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.436268 9965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 12:20:28.439019 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.440243 9965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 12:20:28.440382 9965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "4c259ef22d0d46da9fbe9cdcabe63f2a"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "7e2b3f2f712c674ca94e679ee288d755"
server_key_iv: "1aeecbd942ba1086543068715d0b1512"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.440532 9965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:28.467092 9965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:28.467754 9965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:28.467902 9965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:28.468417 9965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:28.469431 9965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:28.469478 9965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.469518 9965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:28.469548 9965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.478680 9965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38111
I20260620 12:20:28.478744 10092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38111 every 8 connection(s)
I20260620 12:20:28.479669 9965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:20:28.488511 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.481544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36039 (local address 127.0.98.62:38417)
0620 12:20:28.481690 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:28.481694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:28.482720 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 12:20:28.484091 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:28.484094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:28.484138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:28.484211 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:28.485130 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.485653 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.486815 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.487023 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.487095 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:28.488115 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:28.488118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:28.488120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:20:28.488132 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:28.488151 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:20:28.488158 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:28.488216 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:28.488348 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 12:20:28.488399 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:20:28.489379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 9965
I20260620 12:20:28.489400 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.481936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:36039)
0620 12:20:28.482579 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:28.482617 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:28.483910 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:28.484407 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:28.484414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:28.484495 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:28.484978 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:28.484991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.485779 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.485782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:28.486708 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:28.486722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:28.487142 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:28.487145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:28.487280 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:28.487853 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:20:28.487855 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:28.487911 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:20:28.487973 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:28.488229 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:28.488235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:28.488239 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:20:28.488329 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:20:28.488575 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260620 12:20:28.489465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:20:28.489768 392 external_mini_cluster.cc:1428] Setting key 540115055b064d6683644db4c8a2fd7f
I20260620 12:20:28.490590 10093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260620 12:20:28.490828 10093 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:28.491325 10093 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260620 12:20:28.492436 9643 ts_manager.cc:194] Registered new tserver with Master: 4c259ef22d0d46da9fbe9cdcabe63f2a (127.0.98.3:38111)
I20260620 12:20:28.493005 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36039
I20260620 12:20:28.503593 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:20:28.506516 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:28.505515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.0.98.62:38417)
0620 12:20:28.505667 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:28.505672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:28.505839 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:20:28.505994 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:28.505997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:28.506051 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:28.506137 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:28.506383 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39834: BlockingRecv error: recv got EOF from 127.0.0.1:39834 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:20:28.506632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 9703
I20260620 12:20:28.512282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 9834
I20260620 12:20:28.517601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 9965
I20260620 12:20:28.522773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 9612
2026-06-20T12:20:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (873 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T12:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:20:28Z Disabled control of system clock
I20260620 12:20:28.552403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33097
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 12:20:28.657974 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:28.658211 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:28.658245 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:28.662055 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:20:28.662097 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:28.662112 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:28.662129 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:20:28.662143 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:20:28.666671 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:28.667734 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:28.669133 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:28.677166 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.677166 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.677175 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:28.677493 10104 server_base.cc:1034] running on GCE node
I20260620 12:20:28.678009 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:28.679065 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:28.680269 10104 hybrid_clock.cc:648] HybridClock initialized: now 1781958028680258 us; error 37 us; skew 500 ppm
I20260620 12:20:28.686894 10104 webserver.cc:460] Webserver started at http://127.0.98.62:41367/ using document root <none> and password file <none>
I20260620 12:20:28.687498 10104 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:28.687615 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:28.687839 10104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:28.689898 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "8ee18767ecc4485aa7aa013e3ffd3c49"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "e4111881a668d6c573eccaebfdb52c6f"
server_key_iv: "d5579038452a8e6c1978516052ea95f7"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.690442 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ee18767ecc4485aa7aa013e3ffd3c49"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "e4111881a668d6c573eccaebfdb52c6f"
server_key_iv: "d5579038452a8e6c1978516052ea95f7"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.693984 10104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:20:28.696213 10118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.697448 10104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:20:28.697593 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "8ee18767ecc4485aa7aa013e3ffd3c49"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "e4111881a668d6c573eccaebfdb52c6f"
server_key_iv: "d5579038452a8e6c1978516052ea95f7"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.697698 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:28.702077 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:28.702627 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:28.702796 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:28.708901 10104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33097
I20260620 12:20:28.708922 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33097 every 8 connection(s)
I20260620 12:20:28.709800 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:20:28.712729 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:28.717777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 10104
I20260620 12:20:28.717900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:20:28.717870 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49: Bootstrap starting.
I20260620 12:20:28.718243 392 external_mini_cluster.cc:1428] Setting key ce3b32ab8c42fcef59c6e0c1d79f0645
I20260620 12:20:28.720830 10171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:28.721655 10171 log.cc:824] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:28.723541 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49: No bootstrap required, opened a new log
I20260620 12:20:28.724344 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.720036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.0.98.62:33097)
0620 12:20:28.720906 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:28.720928 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:28.720969 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:20:28.721690 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:28.721708 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:28.722051 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:28.722373 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:28.723119 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:28.723124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:28.723136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:20:28.723146 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:28.723197 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:20:28.723212 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:28.723325 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:28.723423 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 12:20:28.723690 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":132,"threads_started":1}
I20260620 12:20:28.727344 10171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee18767ecc4485aa7aa013e3ffd3c49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33097 } }
I20260620 12:20:28.727541 10171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:28.727617 10171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee18767ecc4485aa7aa013e3ffd3c49, State: Initialized, Role: FOLLOWER
I20260620 12:20:28.728032 10171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee18767ecc4485aa7aa013e3ffd3c49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33097 } }
I20260620 12:20:28.728168 10171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:20:28.728227 10171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:20:28.728325 10171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:28.729194 10171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee18767ecc4485aa7aa013e3ffd3c49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33097 } }
I20260620 12:20:28.729507 10171 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee18767ecc4485aa7aa013e3ffd3c49; no voters:
I20260620 12:20:28.729789 10171 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:20:28.729905 10176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:28.730141 10176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee18767ecc4485aa7aa013e3ffd3c49, State: Running, Role: LEADER
I20260620 12:20:28.730424 10176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee18767ecc4485aa7aa013e3ffd3c49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33097 } }
I20260620 12:20:28.730993 10171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:20:28.731971 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee18767ecc4485aa7aa013e3ffd3c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee18767ecc4485aa7aa013e3ffd3c49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33097 } } }
I20260620 12:20:28.732013 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee18767ecc4485aa7aa013e3ffd3c49. Latest consensus state: current_term: 1 leader_uuid: "8ee18767ecc4485aa7aa013e3ffd3c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee18767ecc4485aa7aa013e3ffd3c49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33097 } } }
I20260620 12:20:28.732111 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:28.732129 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:28.732435 10185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:20:28.735409 10185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:20:28.740723 10185 catalog_manager.cc:1347] Generated new cluster ID: 114f68cfb67544a88acd18c777bee5c7
I20260620 12:20:28.740801 10185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:20:28.751818 10185 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:20:28.752120 10185 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:20:28.760457 10185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee18767ecc4485aa7aa013e3ffd3c49: Generated new TSK 0
I20260620 12:20:28.761129 10185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:20:28.769274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33097
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 12:20:28.884177 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:28.884428 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:28.884495 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:28.888616 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:28.888674 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:28.888754 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:20:28.893546 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:28.894606 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:28.895951 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:28.905262 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.905277 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:28.905304 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:28.905591 10195 server_base.cc:1034] running on GCE node
I20260620 12:20:28.905964 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:28.906622 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:28.907779 10195 hybrid_clock.cc:648] HybridClock initialized: now 1781958028907765 us; error 26 us; skew 500 ppm
I20260620 12:20:28.909591 10195 webserver.cc:460] Webserver started at http://127.0.98.1:38655/ using document root <none> and password file <none>
I20260620 12:20:28.910153 10195 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:28.910213 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:28.910459 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:28.912488 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "4583747010fa482d900f5e731caffb6f"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "7b048c665628d979ebb4a05521449751"
server_key_iv: "8533931e07907fe4712e303350a3a7bb"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.913118 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4583747010fa482d900f5e731caffb6f"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "7b048c665628d979ebb4a05521449751"
server_key_iv: "8533931e07907fe4712e303350a3a7bb"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.916882 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 12:20:28.919273 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.920511 10195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:28.920742 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "4583747010fa482d900f5e731caffb6f"
format_stamp: "Formatted at 2026-06-20 12:20:28 on dist-test-slave-9b3m"
server_key: "7b048c665628d979ebb4a05521449751"
server_key_iv: "8533931e07907fe4712e303350a3a7bb"
server_key_version: "encryptionkey@0"
I20260620 12:20:28.920861 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:28.925333 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:28.925915 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:28.926097 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:28.926628 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:28.927526 10195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:28.927595 10195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.927665 10195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:28.927708 10195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:28.936333 10195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35321
I20260620 12:20:28.936419 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35321 every 8 connection(s)
I20260620 12:20:28.937245 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:20:28.943315 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.939022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38637 (local address 127.0.98.62:33097)
0620 12:20:28.939163 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:28.939166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:28.940181 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 12:20:28.941562 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:28.941565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:28.941609 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:28.941718 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:28.942848 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:28.942851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:28.942854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:20:28.942865 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:28.942888 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:20:28.942904 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:28.943006 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:28.943141 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 12:20:28.943183 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:20:28.944121 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:28.939384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33097 (local address 127.0.98.1:38637)
0620 12:20:28.940037 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:28.940075 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:28.941343 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:28.941882 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:28.941888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:28.941983 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:28.942594 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:20:28.942595 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:28.942635 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:20:28.942702 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:28.942991 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:28.942999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:28.943004 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:20:28.943182 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 12:20:28.943410 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
I20260620 12:20:28.945302 10323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33097
I20260620 12:20:28.945504 10323 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:28.945716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 10195
I20260620 12:20:28.945828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:20:28.945948 10323 heartbeater.cc:511] Master 127.0.98.62:33097 requested a full tablet report, sending...
I20260620 12:20:28.946148 392 external_mini_cluster.cc:1428] Setting key 512ea64c7c02f353c19e8a7f0b6ebd7b
I20260620 12:20:28.947589 10134 ts_manager.cc:194] Registered new tserver with Master: 4583747010fa482d900f5e731caffb6f (127.0.98.1:35321)
I20260620 12:20:28.948081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33097
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 12:20:29.057014 10326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:29.057269 10326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:29.057308 10326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:29.061177 10326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:29.061218 10326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:29.061271 10326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:20:29.065781 10326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:29.066725 10326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:29.067984 10326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:29.077693 10332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:29.077718 10331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:29.077986 10326 server_base.cc:1034] running on GCE node
W20260620 12:20:29.077723 10334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:29.078384 10326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:29.079005 10326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:29.080199 10326 hybrid_clock.cc:648] HybridClock initialized: now 1781958029080186 us; error 27 us; skew 500 ppm
I20260620 12:20:29.082048 10326 webserver.cc:460] Webserver started at http://127.0.98.2:35987/ using document root <none> and password file <none>
I20260620 12:20:29.082612 10326 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:29.082701 10326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:29.082934 10326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:29.084926 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "5a61b48ae0174d0591601548141f9704"
format_stamp: "Formatted at 2026-06-20 12:20:29 on dist-test-slave-9b3m"
server_key: "5de43530d4a7b3b4fec382773b1a732c"
server_key_iv: "780dd0d4b1b8325f8a9f0f0154f55547"
server_key_version: "encryptionkey@0"
I20260620 12:20:29.085527 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5a61b48ae0174d0591601548141f9704"
format_stamp: "Formatted at 2026-06-20 12:20:29 on dist-test-slave-9b3m"
server_key: "5de43530d4a7b3b4fec382773b1a732c"
server_key_iv: "780dd0d4b1b8325f8a9f0f0154f55547"
server_key_version: "encryptionkey@0"
I20260620 12:20:29.088956 10326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 12:20:29.091292 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:29.092465 10326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:20:29.092599 10326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "5a61b48ae0174d0591601548141f9704"
format_stamp: "Formatted at 2026-06-20 12:20:29 on dist-test-slave-9b3m"
server_key: "5de43530d4a7b3b4fec382773b1a732c"
server_key_iv: "780dd0d4b1b8325f8a9f0f0154f55547"
server_key_version: "encryptionkey@0"
I20260620 12:20:29.092703 10326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:29.097025 10326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:29.097620 10326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:29.097790 10326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:29.098294 10326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:29.099141 10326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:29.099210 10326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:29.099274 10326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:29.099326 10326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:29.107869 10326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36723
I20260620 12:20:29.107892 10453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36723 every 8 connection(s)
I20260620 12:20:29.108737 10326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:20:29.113929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 10326
I20260620 12:20:29.114099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:20:29.114406 392 external_mini_cluster.cc:1428] Setting key 77ce1f1afe8d999ed4e9a85d11305906
I20260620 12:20:29.114912 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:29.110460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47763 (local address 127.0.98.62:33097)
0620 12:20:29.110617 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:29.110620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:29.111618 (+ 998us) server_negotiation.cc:407] Connection header received
0620 12:20:29.112932 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:29.112935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:29.112987 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:29.113172 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:29.114446 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:29.114450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:29.114453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:20:29.114466 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:29.114491 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:20:29.114503 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:29.114603 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:29.114749 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 12:20:29.114795 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:20:29.115842 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:29.110814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33097 (local address 127.0.98.2:47763)
0620 12:20:29.111469 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:29.111507 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:29.112727 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:29.113329 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:29.113336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:29.113422 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:29.114120 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:20:29.114132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:29.114210 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:20:29.114291 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:29.114606 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:29.114614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:29.114618 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:20:29.114759 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 12:20:29.115023 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260620 12:20:29.116513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33097
--builtin_ntp_servers=127.0.98.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 12:20:29.117031 10454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33097
I20260620 12:20:29.117278 10454 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:29.117666 10454 heartbeater.cc:511] Master 127.0.98.62:33097 requested a full tablet report, sending...
I20260620 12:20:29.118640 10134 ts_manager.cc:194] Registered new tserver with Master: 5a61b48ae0174d0591601548141f9704 (127.0.98.2:36723)
W20260620 12:20:29.226351 10457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:29.226579 10457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:29.226616 10457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:29.230613 10457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:29.230660 10457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:29.230743 10457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:20:29.236178 10457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:29.237258 10457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:29.238596 10457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:29.248157 10462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:29.248175 10457 server_base.cc:1034] running on GCE node
W20260620 12:20:29.248199 10463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:29.248199 10465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:29.248845 10457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:29.249537 10457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:29.250725 10457 hybrid_clock.cc:648] HybridClock initialized: now 1781958029250709 us; error 33 us; skew 500 ppm
I20260620 12:20:29.252525 10457 webserver.cc:460] Webserver started at http://127.0.98.3:42539/ using document root <none> and password file <none>
I20260620 12:20:29.253145 10457 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:29.253234 10457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:29.253469 10457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:29.255461 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "cd958970a0534be2af916bcb4a889cff"
format_stamp: "Formatted at 2026-06-20 12:20:29 on dist-test-slave-9b3m"
server_key: "20149f1248d8dfa74415717af0d661ff"
server_key_iv: "65be75a330f4b3c419983c7b6d46efd8"
server_key_version: "encryptionkey@0"
I20260620 12:20:29.256009 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cd958970a0534be2af916bcb4a889cff"
format_stamp: "Formatted at 2026-06-20 12:20:29 on dist-test-slave-9b3m"
server_key: "20149f1248d8dfa74415717af0d661ff"
server_key_iv: "65be75a330f4b3c419983c7b6d46efd8"
server_key_version: "encryptionkey@0"
I20260620 12:20:29.259588 10457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:20:29.262159 10471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:29.263389 10457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:29.263540 10457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "cd958970a0534be2af916bcb4a889cff"
format_stamp: "Formatted at 2026-06-20 12:20:29 on dist-test-slave-9b3m"
server_key: "20149f1248d8dfa74415717af0d661ff"
server_key_iv: "65be75a330f4b3c419983c7b6d46efd8"
server_key_version: "encryptionkey@0"
I20260620 12:20:29.263648 10457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:29.268087 10457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:29.268627 10457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:29.268808 10457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:29.269387 10457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:29.270210 10457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:29.270282 10457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:29.270344 10457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:29.270393 10457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:29.279345 10457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35877
I20260620 12:20:29.279397 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35877 every 8 connection(s)
I20260620 12:20:29.280154 10457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:20:29.282829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 10457
I20260620 12:20:29.282948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:20:29.283241 392 external_mini_cluster.cc:1428] Setting key 0a3eb53862f2f58d6e3f5b50dafc4bd5
I20260620 12:20:29.286121 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:29.281935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44511 (local address 127.0.98.62:33097)
0620 12:20:29.282106 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:29.282111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:29.283162 (+ 1051us) server_negotiation.cc:407] Connection header received
0620 12:20:29.284487 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:29.284490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:29.284531 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:29.284614 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:29.285691 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:29.285694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:29.285696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:20:29.285706 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:29.285725 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:20:29.285733 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:29.285815 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:29.285977 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 12:20:29.286025 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:20:29.286842 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:29.282344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33097 (local address 127.0.98.3:44511)
0620 12:20:29.283015 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:29.283066 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:29.284282 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:29.284783 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:29.284790 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:29.284874 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:29.285460 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:20:29.285461 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:29.285497 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:20:29.285558 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:29.285792 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:29.285796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:29.285799 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 12:20:29.285911 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 12:20:29.286082 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":135,"threads_started":1}
I20260620 12:20:29.287992 10585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33097
I20260620 12:20:29.288221 10585 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:29.288621 10585 heartbeater.cc:511] Master 127.0.98.62:33097 requested a full tablet report, sending...
I20260620 12:20:29.289630 10134 ts_manager.cc:194] Registered new tserver with Master: cd958970a0534be2af916bcb4a889cff (127.0.98.3:35877)
I20260620 12:20:29.297086 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 12:20:29.300043 10174 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:20:29.299059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.0.98.62:33097)
0620 12:20:29.299211 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:29.299216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:29.299422 (+ 206us) server_negotiation.cc:407] Connection header received
0620 12:20:29.299566 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:29.299569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:29.299603 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:29.299701 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:29.299927 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38656: BlockingRecv error: recv got EOF from 127.0.0.1:38656 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:20:29.300485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 10195
I20260620 12:20:29.306480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 10326
I20260620 12:20:29.311203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 10457
I20260620 12:20:29.316255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 10104
2026-06-20T12:20:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (792 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:20:29 dist-test-slave-9b3m krb5kdc[10595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:20:29 dist-test-slave-9b3m krb5kdc[10595](info): set up 2 sockets
Jun 20 12:20:29 dist-test-slave-9b3m krb5kdc[10595](info): commencing operation
krb5kdc: starting...
W20260620 12:20:30.394011 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:20:30 dist-test-slave-9b3m krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:20:30.516412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37491
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:30.624931 10608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:30.625244 10608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:30.625288 10608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:30.629230 10608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:20:30.629280 10608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:30.629299 10608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:30.629315 10608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:20:30.629330 10608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:20:30.634339 10608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:30.635341 10608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:30.636323 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:30.644654 10614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:30.644675 10613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:30.644748 10616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:30.644868 10608 server_base.cc:1034] running on GCE node
I20260620 12:20:30.645529 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:30.646548 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:30.647729 10608 hybrid_clock.cc:648] HybridClock initialized: now 1781958030647712 us; error 29 us; skew 500 ppm
Jun 20 12:20:30 dist-test-slave-9b3m krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:30.650310 10608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:20:30.651168 10608 webserver.cc:460] Webserver started at http://127.0.98.62:44029/ using document root <none> and password file <none>
I20260620 12:20:30.651702 10608 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:30.651791 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:30.652016 10608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:30.654003 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "a845af16f27d46c78691d4cc47e990b6"
format_stamp: "Formatted at 2026-06-20 12:20:30 on dist-test-slave-9b3m"
server_key: "8296a0c3c72e30db44aa40b70334ba9b"
server_key_iv: "9601a7622487dbc36e69b35fd7fd6ca8"
server_key_version: "encryptionkey@0"
I20260620 12:20:30.654537 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "a845af16f27d46c78691d4cc47e990b6"
format_stamp: "Formatted at 2026-06-20 12:20:30 on dist-test-slave-9b3m"
server_key: "8296a0c3c72e30db44aa40b70334ba9b"
server_key_iv: "9601a7622487dbc36e69b35fd7fd6ca8"
server_key_version: "encryptionkey@0"
I20260620 12:20:30.657976 10608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:20:30.660190 10623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:30.661381 10608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:20:30.661513 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "a845af16f27d46c78691d4cc47e990b6"
format_stamp: "Formatted at 2026-06-20 12:20:30 on dist-test-slave-9b3m"
server_key: "8296a0c3c72e30db44aa40b70334ba9b"
server_key_iv: "9601a7622487dbc36e69b35fd7fd6ca8"
server_key_version: "encryptionkey@0"
I20260620 12:20:30.661617 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:30.671638 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:30.675387 10608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:30.675582 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:30.682320 10608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37491
I20260620 12:20:30.682605 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37491 every 8 connection(s)
I20260620 12:20:30.683252 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:20:30.685940 10676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:30.691028 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6: Bootstrap starting.
I20260620 12:20:30.691958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 10608
I20260620 12:20:30.692087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:20:30.692349 392 external_mini_cluster.cc:1428] Setting key a8bc8ae9ed041af16e806a9d291e90b1
I20260620 12:20:30.693608 10676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:30.694329 10676 log.cc:824] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:30.696058 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6: No bootstrap required, opened a new log
I20260620 12:20:30.699422 10676 raft_consensus.cc:348] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a845af16f27d46c78691d4cc47e990b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37491 } }
I20260620 12:20:30.699613 10676 raft_consensus.cc:374] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:30.699682 10676 raft_consensus.cc:729] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845af16f27d46c78691d4cc47e990b6, State: Initialized, Role: FOLLOWER
I20260620 12:20:30.700120 10676 consensus_queue.cc:260] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a845af16f27d46c78691d4cc47e990b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37491 } }
I20260620 12:20:30.700271 10676 raft_consensus.cc:388] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:20:30.700353 10676 raft_consensus.cc:482] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:20:30.700476 10676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 12:20:30 dist-test-slave-9b3m krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:20:30.701548 10676 raft_consensus.cc:504] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a845af16f27d46c78691d4cc47e990b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37491 } }
I20260620 12:20:30.701884 10676 leader_election.cc:302] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a845af16f27d46c78691d4cc47e990b6; no voters:
I20260620 12:20:30.702178 10676 leader_election.cc:288] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:20:30.702328 10681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:30.702540 10681 raft_consensus.cc:686] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [term 1 LEADER]: Becoming Leader. State: Replica: a845af16f27d46c78691d4cc47e990b6, State: Running, Role: LEADER
I20260620 12:20:30.702839 10681 consensus_queue.cc:237] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a845af16f27d46c78691d4cc47e990b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37491 } }
I20260620 12:20:30.703402 10676 sys_catalog.cc:564] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:20:30.703992 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a845af16f27d46c78691d4cc47e990b6. Latest consensus state: current_term: 1 leader_uuid: "a845af16f27d46c78691d4cc47e990b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a845af16f27d46c78691d4cc47e990b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37491 } } }
I20260620 12:20:30.704101 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:30.704725 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a845af16f27d46c78691d4cc47e990b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a845af16f27d46c78691d4cc47e990b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37491 } } }
I20260620 12:20:30.704818 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6 [sys.catalog]: This master's current role is: LEADER
I20260620 12:20:30.705758 10687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:20:30.707047 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:30.693408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.0.98.62:37491)
0620 12:20:30.693884 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:30.693899 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:30.693933 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:20:30.694634 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:30.694651 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:30.694959 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:30.695287 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:30.696031 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:30.697022 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:30.697885 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:30.698367 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:30.698532 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:30.701193 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:30.701198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:30.701210 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:30.701234 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:30.703244 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:30.703734 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:30.703740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:30.703746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:30.703886 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:30.704323 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:30.704327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:30.704329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:30.704958 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:30.705759 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:30.705910 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 12:20:30.706178 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":141,"threads_started":1}
I20260620 12:20:30.708330 10687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:20:30.714166 10687 catalog_manager.cc:1347] Generated new cluster ID: 8d72583980d748139445cc47cb679033
I20260620 12:20:30.714228 10687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:20:30.732139 10687 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:20:30.733114 10687 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:20:30.743130 10687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6: Generated new TSK 0
I20260620 12:20:30.743856 10687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:20:30.772864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37491
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:30.880371 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:30.880692 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:30.880784 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:30.885516 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:30.885605 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:30.885700 10702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:20:30.890980 10702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:30.892045 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:30.893100 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:30.902841 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:30.902825 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:30.902832 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:30.902890 10702 server_base.cc:1034] running on GCE node
I20260620 12:20:30.903424 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:30.904098 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:30.905273 10702 hybrid_clock.cc:648] HybridClock initialized: now 1781958030905241 us; error 50 us; skew 500 ppm
Jun 20 12:20:30 dist-test-slave-9b3m krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:30.907883 10702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:20:30.908753 10702 webserver.cc:460] Webserver started at http://127.0.98.1:40603/ using document root <none> and password file <none>
I20260620 12:20:30.909334 10702 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:30.909395 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:30.909552 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:30.911455 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a905b2a0ff34663bc6d09639a01c58d"
format_stamp: "Formatted at 2026-06-20 12:20:30 on dist-test-slave-9b3m"
server_key: "86251ba5bc0d6dccf5a90d0eb14a03a2"
server_key_iv: "567eaaddd9e5a269d5f29abad71442fb"
server_key_version: "encryptionkey@0"
I20260620 12:20:30.911948 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a905b2a0ff34663bc6d09639a01c58d"
format_stamp: "Formatted at 2026-06-20 12:20:30 on dist-test-slave-9b3m"
server_key: "86251ba5bc0d6dccf5a90d0eb14a03a2"
server_key_iv: "567eaaddd9e5a269d5f29abad71442fb"
server_key_version: "encryptionkey@0"
I20260620 12:20:30.915753 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:20:30.918380 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:30.919726 10702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:20:30.919864 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "9a905b2a0ff34663bc6d09639a01c58d"
format_stamp: "Formatted at 2026-06-20 12:20:30 on dist-test-slave-9b3m"
server_key: "86251ba5bc0d6dccf5a90d0eb14a03a2"
server_key_iv: "567eaaddd9e5a269d5f29abad71442fb"
server_key_version: "encryptionkey@0"
I20260620 12:20:30.919981 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:30.940973 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:30.943889 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:30.944099 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:30.944726 10702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:30.945667 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:30.945716 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:30.945801 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:30.945835 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:30.955493 10702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34011
I20260620 12:20:30.955514 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34011 every 8 connection(s)
I20260620 12:20:30.956424 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:20:30.958690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 10702
I20260620 12:20:30.958820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:20:30.959093 392 external_mini_cluster.cc:1428] Setting key ac0f318f962747e6df8327249b602988
Jun 20 12:20:30 dist-test-slave-9b3m krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:20:30.971133 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:30.958309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34447 (local address 127.0.98.62:37491)
0620 12:20:30.958466 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:30.958473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:30.959641 (+ 1168us) server_negotiation.cc:407] Connection header received
0620 12:20:30.960696 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:30.960699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:30.960747 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:30.960823 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:30.962012 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:30.962536 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:30.963670 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:30.963903 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:30.964037 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:30.967084 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:30.967089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:30.967092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:30.967122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:30.968864 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:30.969473 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:30.969476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:30.969477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:30.969556 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:30.969885 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:30.969888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:30.969889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:30.970058 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:30.970134 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:30.970756 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 12:20:30.970929 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:20:30.971984 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:30.958685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37491 (local address 127.0.98.1:34447)
0620 12:20:30.959476 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:30.959526 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:30.960487 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:30.960991 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:30.960997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:30.961411 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:30.961871 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:30.961884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:30.962658 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:30.962662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:30.963547 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:30.963556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:30.964071 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:30.964075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:30.964250 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:30.964944 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:30.964961 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:30.966845 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:30.968993 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:30.969001 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:30.969016 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:30.969364 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:30.969650 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:30.969652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:30.969653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:30.969791 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:30.970187 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:30.970226 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:30.970500 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 12:20:30.970699 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 12:20:30.971024 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":184,"threads_started":1}
I20260620 12:20:30.973403 10831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37491
I20260620 12:20:30.973646 10831 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:30.974128 10831 heartbeater.cc:511] Master 127.0.98.62:37491 requested a full tablet report, sending...
I20260620 12:20:30.975701 10640 ts_manager.cc:194] Registered new tserver with Master: 9a905b2a0ff34663bc6d09639a01c58d (127.0.98.1:34011)
I20260620 12:20:30.976800 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:20:30.988849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37491
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:31.098484 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:31.098731 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:31.098775 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:31.102777 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:31.102842 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:31.102941 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:20:31.108345 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:31.109452 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:31.110471 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:20:31.120249 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:31.120249 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:31.120249 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:31.120515 10836 server_base.cc:1034] running on GCE node
I20260620 12:20:31.121173 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:31.121855 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:31.123049 10836 hybrid_clock.cc:648] HybridClock initialized: now 1781958031123031 us; error 32 us; skew 500 ppm
Jun 20 12:20:31 dist-test-slave-9b3m krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:31.126279 10836 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:20:31.127259 10836 webserver.cc:460] Webserver started at http://127.0.98.2:44409/ using document root <none> and password file <none>
I20260620 12:20:31.127847 10836 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:31.127907 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:31.128147 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:31.130263 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "c00deec85308497d9c43fa180f53058d"
format_stamp: "Formatted at 2026-06-20 12:20:31 on dist-test-slave-9b3m"
server_key: "4cd32c98f023ffc6b45deb69fcf99f34"
server_key_iv: "bf383858896030c4d193b59fc56f890c"
server_key_version: "encryptionkey@0"
I20260620 12:20:31.130823 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c00deec85308497d9c43fa180f53058d"
format_stamp: "Formatted at 2026-06-20 12:20:31 on dist-test-slave-9b3m"
server_key: "4cd32c98f023ffc6b45deb69fcf99f34"
server_key_iv: "bf383858896030c4d193b59fc56f890c"
server_key_version: "encryptionkey@0"
I20260620 12:20:31.134608 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 12:20:31.137177 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:31.138402 10836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:20:31.138536 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "c00deec85308497d9c43fa180f53058d"
format_stamp: "Formatted at 2026-06-20 12:20:31 on dist-test-slave-9b3m"
server_key: "4cd32c98f023ffc6b45deb69fcf99f34"
server_key_iv: "bf383858896030c4d193b59fc56f890c"
server_key_version: "encryptionkey@0"
I20260620 12:20:31.138665 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:31.150024 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:31.152755 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:31.152956 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:31.153551 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:31.154331 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:31.154402 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:31.154469 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:31.154522 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:31.163075 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35121
I20260620 12:20:31.163138 10964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35121 every 8 connection(s)
I20260620 12:20:31.163956 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:20:31.165246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 10836
I20260620 12:20:31.165376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:20:31.165640 392 external_mini_cluster.cc:1428] Setting key 66f906b2da09d5ec9e77c143d6d3b51e
Jun 20 12:20:31 dist-test-slave-9b3m krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 12:20:31.179282 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.165801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46621 (local address 127.0.98.62:37491)
0620 12:20:31.165959 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.165963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.167365 (+ 1402us) server_negotiation.cc:407] Connection header received
0620 12:20:31.168718 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.168720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.168760 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.168845 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:31.170138 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.171007 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.172348 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.172773 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.172916 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.175467 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:31.175471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:31.175473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:31.175506 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:31.177002 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:31.177600 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:31.177603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:31.177605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:31.177739 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:31.178139 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:31.178146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:31.178151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:31.178345 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:31.178423 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.179063 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.179163 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:20:31.180524 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.166316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37491 (local address 127.0.98.2:46621)
0620 12:20:31.167193 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:31.167251 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:31.168490 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:31.169037 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:31.169087 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:31.169502 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:31.169976 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.169990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.171181 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.171184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.172217 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.172226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.172945 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.172948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.173183 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.173798 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:31.173814 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:31.175301 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:31.177187 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:31.177210 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:31.177228 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:31.177503 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:31.177859 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:31.177862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:31.177863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:31.178043 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:31.178452 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:31.178462 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:31.178799 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 12:20:31.179028 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 12:20:31.179398 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":138,"threads_started":1}
I20260620 12:20:31.181777 10965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37491
I20260620 12:20:31.182058 10965 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:31.182556 10965 heartbeater.cc:511] Master 127.0.98.62:37491 requested a full tablet report, sending...
I20260620 12:20:31.183641 10640 ts_manager.cc:194] Registered new tserver with Master: c00deec85308497d9c43fa180f53058d (127.0.98.2:35121)
I20260620 12:20:31.184213 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:20:31.194273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37491
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:20:31.304697 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:20:31.305095 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:20:31.305179 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:20:31.309844 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:20:31.309932 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:20:31.310029 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:20:31.315428 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:20:31.316586 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:20:31.317713 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:20:31.327685 10970 server_base.cc:1034] running on GCE node
W20260620 12:20:31.327603 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:31.327603 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:20:31.327607 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:20:31.328313 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:20:31.329005 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:20:31.330240 10970 hybrid_clock.cc:648] HybridClock initialized: now 1781958031330209 us; error 54 us; skew 500 ppm
Jun 20 12:20:31 dist-test-slave-9b3m krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:20:31.332996 10970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:20:31.333885 10970 webserver.cc:460] Webserver started at http://127.0.98.3:44237/ using document root <none> and password file <none>
I20260620 12:20:31.334455 10970 fs_manager.cc:359] Metadata directory not provided
I20260620 12:20:31.334547 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:20:31.334779 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:20:31.336774 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d61fa2736a44bdd8e95a721b9de4d37"
format_stamp: "Formatted at 2026-06-20 12:20:31 on dist-test-slave-9b3m"
server_key: "962be57e302328fb7f5eaa10686053fb"
server_key_iv: "578c830d2d122c60aadcef641f0d1ff8"
server_key_version: "encryptionkey@0"
I20260620 12:20:31.337355 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d61fa2736a44bdd8e95a721b9de4d37"
format_stamp: "Formatted at 2026-06-20 12:20:31 on dist-test-slave-9b3m"
server_key: "962be57e302328fb7f5eaa10686053fb"
server_key_iv: "578c830d2d122c60aadcef641f0d1ff8"
server_key_version: "encryptionkey@0"
I20260620 12:20:31.340929 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 12:20:31.343397 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:31.344798 10970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:20:31.344945 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "3d61fa2736a44bdd8e95a721b9de4d37"
format_stamp: "Formatted at 2026-06-20 12:20:31 on dist-test-slave-9b3m"
server_key: "962be57e302328fb7f5eaa10686053fb"
server_key_iv: "578c830d2d122c60aadcef641f0d1ff8"
server_key_version: "encryptionkey@0"
I20260620 12:20:31.345151 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:20:31.361716 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:20:31.365029 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:20:31.365312 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:20:31.365933 10970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:20:31.366827 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:20:31.366904 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:31.366977 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:20:31.367033 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:20:31.376705 10970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35861
I20260620 12:20:31.376744 11098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35861 every 8 connection(s)
I20260620 12:20:31.377642 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:20:31.380476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 10970
I20260620 12:20:31.380587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:20:31.380918 392 external_mini_cluster.cc:1428] Setting key bc01cf541a0902d15574803a424a79d1
Jun 20 12:20:31 dist-test-slave-9b3m krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:20:31.391590 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.379421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42541 (local address 127.0.98.62:37491)
0620 12:20:31.379565 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.379569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.380769 (+ 1200us) server_negotiation.cc:407] Connection header received
0620 12:20:31.381924 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.381927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.381983 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.382065 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:31.383565 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.384166 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.385363 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.385583 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.385680 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.388084 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:31.388087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:31.388089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:31.388114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:31.389863 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:31.390307 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:31.390310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:31.390311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:31.390388 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:31.390692 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:31.390695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:31.390696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:31.390870 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:31.390949 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.391369 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.391466 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:20:31.392398 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.379793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37491 (local address 127.0.98.3:42541)
0620 12:20:31.380546 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:31.380605 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:31.381693 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:31.382308 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:31.382320 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:31.382833 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:20:31.383432 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.383448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.384294 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.384297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.385250 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.385259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.385741 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.385745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.385884 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.386450 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:20:31.386466 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:20:31.387907 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:20:31.389969 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:31.389973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:31.389984 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:31.390217 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:31.390468 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:20:31.390470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:20:31.390471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:20:31.390597 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:20:31.390958 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:20:31.390962 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:20:31.391172 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 12:20:31.391320 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 12:20:31.391548 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":124,"threads_started":1}
I20260620 12:20:31.393605 11099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37491
I20260620 12:20:31.393852 11099 heartbeater.cc:464] Registering TS with master...
I20260620 12:20:31.394400 11099 heartbeater.cc:511] Master 127.0.98.62:37491 requested a full tablet report, sending...
I20260620 12:20:31.395385 10640 ts_manager.cc:194] Registered new tserver with Master: 3d61fa2736a44bdd8e95a721b9de4d37 (127.0.98.3:35861)
I20260620 12:20:31.395965 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42541
I20260620 12:20:31.406599 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:20:31.415737 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.408377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.0.98.62:37491)
0620 12:20:31.408535 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.408540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.408688 (+ 148us) server_negotiation.cc:407] Connection header received
0620 12:20:31.408867 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.408870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.408919 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.409000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:20:31.409665 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.410168 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.410992 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.411212 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.411320 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.412637 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:20:31.412641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:20:31.412643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:20:31.412675 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:20:31.414120 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:31.414533 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:31.414538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:31.414540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:31.414617 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:20:31.414895 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:20:31.414900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:20:31.414901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:20:31.415106 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:20:31.415255 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.415468 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.415578 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:20:31.418757 10640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:20:31.421800 10640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:20:31.435570 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.430171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35121 (local address 127.0.0.1:55702)
0620 12:20:31.430626 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:31.430671 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:31.430791 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:31.431710 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:31.431716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:31.431737 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:31.431881 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.431888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.433127 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.433131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.434267 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.434275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.435035 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.435039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.435142 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.435177 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 12:20:31.435308 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 12:20:31.435425 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260620 12:20:31.435583 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.430296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.98.2:35121)
0620 12:20:31.431210 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.431223 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.431247 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:20:31.431333 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.431338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.431542 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.431721 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:31.432003 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.432972 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.434379 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.434918 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.435038 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.435182 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.435334 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.435425 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":122,"threads_started":1}
I20260620 12:20:31.436771 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.430241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35861 (local address 127.0.0.1:54932)
0620 12:20:31.430851 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:31.430866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:31.430937 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:31.432424 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:31.432427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:31.432443 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:31.432526 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.432531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.433879 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.433882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.435211 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.435218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.436337 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.436340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.436450 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.436484 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:20:31.436565 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:20:31.436631 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":133,"thread_start_us":111,"threads_started":1}
I20260620 12:20:31.436884 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.430957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.0.98.3:35861)
0620 12:20:31.432034 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.432053 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.432069 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:20:31.432123 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.432126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.432297 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.432451 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:31.432600 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.433725 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.435351 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.436209 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.436327 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.436459 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.436606 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.436737 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":107,"threads_started":1}
I20260620 12:20:31.438545 10899 tablet_service.cc:1467] Processing CreateTablet for tablet b81734a5517245acba42cbce0198f594 (DEFAULT_TABLE table=test-table [id=3ca44a60e233466db30f48862ca505b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:31.439108 11033 tablet_service.cc:1467] Processing CreateTablet for tablet b81734a5517245acba42cbce0198f594 (DEFAULT_TABLE table=test-table [id=3ca44a60e233466db30f48862ca505b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:31.439447 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81734a5517245acba42cbce0198f594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:31.439862 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81734a5517245acba42cbce0198f594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:31.441862 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.431277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.0.98.1:34011)
0620 12:20:31.431575 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.431591 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.431779 (+ 188us) server_negotiation.cc:407] Connection header received
0620 12:20:31.431910 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.431913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.432101 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.432245 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:31.432504 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.433725 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.435351 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.440907 (+ 5556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.441023 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.441185 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.441601 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.441703 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":121,"threads_started":1}
I20260620 12:20:31.442039 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.430905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34011 (local address 127.0.0.1:34106)
0620 12:20:31.431678 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:31.431689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:31.431798 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:31.432237 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:31.432240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:31.432257 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:31.432369 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.432377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.433879 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.433882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.435207 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.435218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.441263 (+ 6045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.441266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.441365 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.441398 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 12:20:31.441540 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 12:20:31.441619 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":46,"thread_start_us":81,"threads_started":1}
I20260620 12:20:31.444815 10765 tablet_service.cc:1467] Processing CreateTablet for tablet b81734a5517245acba42cbce0198f594 (DEFAULT_TABLE table=test-table [id=3ca44a60e233466db30f48862ca505b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:20:31.445859 10765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81734a5517245acba42cbce0198f594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:20:31.446386 11116 tablet_bootstrap.cc:492] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: Bootstrap starting.
I20260620 12:20:31.447603 11117 tablet_bootstrap.cc:492] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: Bootstrap starting.
I20260620 12:20:31.448510 11116 tablet_bootstrap.cc:654] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:31.449765 11116 log.cc:824] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:31.450165 11117 tablet_bootstrap.cc:654] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:31.450986 11117 log.cc:824] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:31.451781 11116 tablet_bootstrap.cc:492] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: No bootstrap required, opened a new log
I20260620 12:20:31.451979 11116 ts_tablet_manager.cc:1397] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:20:31.452818 11120 tablet_bootstrap.cc:492] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d: Bootstrap starting.
I20260620 12:20:31.454905 11120 tablet_bootstrap.cc:654] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d: Neither blocks nor log segments found. Creating new log.
I20260620 12:20:31.456106 11120 log.cc:824] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:20:31.456099 11116 raft_consensus.cc:348] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.456386 11116 raft_consensus.cc:374] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:31.456455 11116 raft_consensus.cc:729] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d61fa2736a44bdd8e95a721b9de4d37, State: Initialized, Role: FOLLOWER
I20260620 12:20:31.456502 11117 tablet_bootstrap.cc:492] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: No bootstrap required, opened a new log
I20260620 12:20:31.456696 11117 ts_tablet_manager.cc:1397] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 12:20:31.456910 11116 consensus_queue.cc:260] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.457599 11099 heartbeater.cc:503] Master 127.0.98.62:37491 was elected leader, sending a full tablet report...
I20260620 12:20:31.458051 11116 ts_tablet_manager.cc:1428] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:20:31.458952 11120 tablet_bootstrap.cc:492] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d: No bootstrap required, opened a new log
I20260620 12:20:31.459126 11120 ts_tablet_manager.cc:1397] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:20:31.460187 11117 raft_consensus.cc:348] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.460453 11117 raft_consensus.cc:374] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:31.460513 11117 raft_consensus.cc:729] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00deec85308497d9c43fa180f53058d, State: Initialized, Role: FOLLOWER
I20260620 12:20:31.460949 11117 consensus_queue.cc:260] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.461621 10965 heartbeater.cc:503] Master 127.0.98.62:37491 was elected leader, sending a full tablet report...
I20260620 12:20:31.461946 11117 ts_tablet_manager.cc:1428] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:20:31.462829 11120 raft_consensus.cc:348] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.463100 11120 raft_consensus.cc:374] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:20:31.463160 11120 raft_consensus.cc:729] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a905b2a0ff34663bc6d09639a01c58d, State: Initialized, Role: FOLLOWER
I20260620 12:20:31.463567 11120 consensus_queue.cc:260] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.464164 10831 heartbeater.cc:503] Master 127.0.98.62:37491 was elected leader, sending a full tablet report...
I20260620 12:20:31.464425 11120 ts_tablet_manager.cc:1428] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:20:31.524605 11124 raft_consensus.cc:482] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:20:31.524780 11124 raft_consensus.cc:504] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.525915 11124 leader_election.cc:288] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c00deec85308497d9c43fa180f53058d (127.0.98.2:35121), 3d61fa2736a44bdd8e95a721b9de4d37 (127.0.98.3:35861)
I20260620 12:20:31.529748 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.526278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54871 (local address 127.0.98.3:35861)
0620 12:20:31.526405 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.526410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.526577 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:20:31.526724 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.526727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.526782 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.526866 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:31.527131 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.527624 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.528724 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.529167 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.529296 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.529363 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.529559 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.529625 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:20:31.529764 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.526198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51491 (local address 127.0.98.2:35121)
0620 12:20:31.526341 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.526348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.526526 (+ 178us) server_negotiation.cc:407] Connection header received
0620 12:20:31.526664 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.526672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.526717 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.526809 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:20:31.527088 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.527580 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.528727 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.529199 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.529309 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.529368 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.529578 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.529644 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:20:31.529779 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.526071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35121 (local address 127.0.98.1:51491)
0620 12:20:31.526421 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:31.526436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:31.526571 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:31.526816 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:31.526820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:31.526834 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:31.526970 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.526976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.527686 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.527691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.528606 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.528615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.529312 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.529315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.529455 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.529472 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 12:20:31.529563 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:20:31.529631 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":91,"thread_start_us":95,"threads_started":1}
I20260620 12:20:31.529786 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.526171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35861 (local address 127.0.98.1:54871)
0620 12:20:31.526509 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:31.526521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:20:31.526607 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:20:31.526878 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:20:31.526881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:20:31.526896 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:20:31.527022 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.527028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.527736 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.527739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.528598 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:20:31.528607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.529297 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.529300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:20:31.529437 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.529455 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:20:31.529544 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 12:20:31.529613 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":25,"thread_start_us":144,"threads_started":1}
I20260620 12:20:31.530409 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81734a5517245acba42cbce0198f594" candidate_uuid: "9a905b2a0ff34663bc6d09639a01c58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" is_pre_election: true
I20260620 12:20:31.530419 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81734a5517245acba42cbce0198f594" candidate_uuid: "9a905b2a0ff34663bc6d09639a01c58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c00deec85308497d9c43fa180f53058d" is_pre_election: true
I20260620 12:20:31.530682 11053 raft_consensus.cc:2454] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a905b2a0ff34663bc6d09639a01c58d in term 0.
I20260620 12:20:31.530691 10919 raft_consensus.cc:2454] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a905b2a0ff34663bc6d09639a01c58d in term 0.
I20260620 12:20:31.531103 10719 leader_election.cc:302] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d61fa2736a44bdd8e95a721b9de4d37, 9a905b2a0ff34663bc6d09639a01c58d; no voters:
I20260620 12:20:31.531358 11124 raft_consensus.cc:2781] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:20:31.531438 11124 raft_consensus.cc:482] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:20:31.531491 11124 raft_consensus.cc:3037] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:31.532505 11124 raft_consensus.cc:504] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.532884 11124 leader_election.cc:288] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [CANDIDATE]: Term 1 election: Requested vote from peers c00deec85308497d9c43fa180f53058d (127.0.98.2:35121), 3d61fa2736a44bdd8e95a721b9de4d37 (127.0.98.3:35861)
I20260620 12:20:31.533175 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81734a5517245acba42cbce0198f594" candidate_uuid: "9a905b2a0ff34663bc6d09639a01c58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c00deec85308497d9c43fa180f53058d"
I20260620 12:20:31.533309 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81734a5517245acba42cbce0198f594" candidate_uuid: "9a905b2a0ff34663bc6d09639a01c58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d61fa2736a44bdd8e95a721b9de4d37"
I20260620 12:20:31.533324 10919 raft_consensus.cc:3037] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:31.533435 11053 raft_consensus.cc:3037] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:20:31.534415 10919 raft_consensus.cc:2454] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a905b2a0ff34663bc6d09639a01c58d in term 1.
I20260620 12:20:31.534482 11053 raft_consensus.cc:2454] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a905b2a0ff34663bc6d09639a01c58d in term 1.
I20260620 12:20:31.534695 10721 leader_election.cc:302] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a905b2a0ff34663bc6d09639a01c58d, c00deec85308497d9c43fa180f53058d; no voters:
I20260620 12:20:31.534855 11124 raft_consensus.cc:2781] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:20:31.535055 11124 raft_consensus.cc:686] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [term 1 LEADER]: Becoming Leader. State: Replica: 9a905b2a0ff34663bc6d09639a01c58d, State: Running, Role: LEADER
I20260620 12:20:31.535395 11124 consensus_queue.cc:237] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } }
I20260620 12:20:31.538406 10640 catalog_manager.cc:5495] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a905b2a0ff34663bc6d09639a01c58d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9a905b2a0ff34663bc6d09639a01c58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a905b2a0ff34663bc6d09639a01c58d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34011 } health_report { overall_health: HEALTHY } } }
I20260620 12:20:31.555837 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.552281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.0.98.1:34011)
0620 12:20:31.552400 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:20:31.552407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:20:31.552486 (+ 79us) server_negotiation.cc:407] Connection header received
0620 12:20:31.552671 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:20:31.552674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:20:31.552723 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:20:31.552828 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:20:31.553117 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.553609 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.554454 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:20:31.554669 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:20:31.554770 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:20:31.555073 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:20:31.555460 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:20:31.555588 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:20:31.555647 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 12:20:31.555699 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:20:31.561235 10919 raft_consensus.cc:1261] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 1 FOLLOWER]: Refusing update from remote peer 9a905b2a0ff34663bc6d09639a01c58d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:31.561285 11053 raft_consensus.cc:1261] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 1 FOLLOWER]: Refusing update from remote peer 9a905b2a0ff34663bc6d09639a01c58d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:20:31.561916 11124 consensus_queue.cc:1035] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c00deec85308497d9c43fa180f53058d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:31.562150 11127 consensus_queue.cc:1035] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d61fa2736a44bdd8e95a721b9de4d37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:20:31.569136 11131 mvcc.cc:205] Tried to move back new op lower bound from 7298900097268264960 to 7298900097171685376. Current Snapshot: MvccSnapshot[applied={T|T < 7298900097268264960}]
I20260620 12:20:31.577946 10640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:31.578125 10640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:20:31.580428 10640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a845af16f27d46c78691d4cc47e990b6: Sending DeleteTablet for 3 replicas of tablet b81734a5517245acba42cbce0198f594
I20260620 12:20:31.581209 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet b81734a5517245acba42cbce0198f594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55702
I20260620 12:20:31.581241 10765 tablet_service.cc:1514] Processing DeleteTablet for tablet b81734a5517245acba42cbce0198f594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34106
I20260620 12:20:31.581274 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet b81734a5517245acba42cbce0198f594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54932
I20260620 12:20:31.581751 11141 tablet_replica.cc:330] T b81734a5517245acba42cbce0198f594 P 9a905b2a0ff34663bc6d09639a01c58d: stopping tablet replica
I20260620 12:20:31.581954 11142 tablet_replica.cc:330] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: stopping tablet replica
I20260620 12:20:31.582126 11140 tablet_replica.cc:330] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: stopping tablet replica
I20260620 12:20:31.582262 11142 raft_consensus.cc:2229] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:31.582405 11140 raft_consensus.cc:2229] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:20:31.582533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 10702
I20260620 12:20:31.582569 11142 raft_consensus.cc:2258] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:31.582679 11140 raft_consensus.cc:2258] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:20:31.585318 11140 ts_tablet_manager.cc:1905] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:31.585688 11142 ts_tablet_manager.cc:1905] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:20:31.587735 11142 ts_tablet_manager.cc:1918] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:31.587855 11142 log.cc:1196] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/b81734a5517245acba42cbce0198f594
I20260620 12:20:31.588143 11142 ts_tablet_manager.cc:1939] T b81734a5517245acba42cbce0198f594 P 3d61fa2736a44bdd8e95a721b9de4d37: Deleting consensus metadata
I20260620 12:20:31.588706 11140 ts_tablet_manager.cc:1918] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:20:31.588840 11140 log.cc:1196] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/b81734a5517245acba42cbce0198f594
I20260620 12:20:31.589202 11140 ts_tablet_manager.cc:1939] T b81734a5517245acba42cbce0198f594 P c00deec85308497d9c43fa180f53058d: Deleting consensus metadata
I20260620 12:20:31.589435 10625 catalog_manager.cc:4841] TS 3d61fa2736a44bdd8e95a721b9de4d37 (127.0.98.3:35861): tablet b81734a5517245acba42cbce0198f594 (table test-table [id=3ca44a60e233466db30f48862ca505b0]) successfully deleted
W20260620 12:20:31.590049 10625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34011 (error 108)
I20260620 12:20:31.590271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 10836
I20260620 12:20:31.590380 10627 catalog_manager.cc:4841] TS c00deec85308497d9c43fa180f53058d (127.0.98.2:35121): tablet b81734a5517245acba42cbce0198f594 (table test-table [id=3ca44a60e233466db30f48862ca505b0]) successfully deleted
I20260620 12:20:31.592247 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:20:31.591981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34011 (local address 127.0.0.1:34122)
0620 12:20:31.592107 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 12:20:31.592176 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 12:20:31.592386 10625 catalog_manager.cc:4584] TS 9a905b2a0ff34663bc6d09639a01c58d (127.0.98.1:34011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b81734a5517245acba42cbce0198f594: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34011: connect: Connection refused (error 111)
I20260620 12:20:31.595989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 10970
I20260620 12:20:31.601431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 10608
2026-06-20T12:20:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2286 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:20:31 dist-test-slave-9b3m krb5kdc[11146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:20:31 dist-test-slave-9b3m krb5kdc[11146](info): set up 2 sockets
Jun 20 12:20:31 dist-test-slave-9b3m krb5kdc[11146](info): commencing operation
krb5kdc: starting...
W20260620 12:20:32.638041 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:20:32 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:20:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 12:20:32.805202 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZkvxte/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/postgres -l logfile start
2026-06-20 12:20:35.229 UTC [11176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 12:20:35.229 UTC [11176] LOG: listening on IPv4 address "127.0.98.20", port 39427
2026-06-20 12:20:35.232 UTC [11176] LOG: listening on Unix socket "/tmp/.s.PGSQL.39427"
2026-06-20 12:20:35.267 UTC [11179] LOG: database system was shut down at 2026-06-20 12:20:33 UTC
2026-06-20 12:20:35.270 UTC [11176] LOG: database system is ready to accept connections
I20260620 12:20:36.209877 392 mini_postgres.cc:96] Postgres bound to 39427
2026-06-20 12:20:36.216 UTC [11187] FATAL: database "slave" does not exist
127.0.98.20:39427 - accepting connections
I20260620 12:20:36.216846 392 mini_ranger.cc:161] Starting Ranger...
I20260620 12:20:36.231643 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 12:20:36.738830 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 12:20:36.738931 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-admin
2026-06-20 12:20:37,033 [I] DB FLAVOR :POSTGRES
2026-06-20 12:20:37,034 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:20:37,034 [I] Checking connection..
2026-06-20 12:20:37,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 12:20:37,300 [I] Checking connection passed.
2026-06-20 12:20:37,300 [I] --------- Verifying version history table ---------
2026-06-20 12:20:37,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:20:37,552 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:20:37,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:20:37,803 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:20:37,803 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 12:20:37,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 12:20:38.049 UTC [11251] WARNING: there is no transaction in progress
2026-06-20 12:20:38,055 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 12:20:38,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:20:38,299 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 12:20:38,299 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:20:38,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 12:20:38,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 12:20:38,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9b3m.c.gcp-upstream.internal','N') ;"
2026-06-20 12:20:39,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:20:39,254 [I] Table x_portal_user does not exist in database ranger
2026-06-20 12:20:39,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:20:39,477 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 12:20:39,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:20:39,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:20:39,932 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 12:20:39,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 12:20:40.894 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:40.911 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:40.928 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:40.944 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:40.960 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:40.979 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.040 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.048 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.064 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.075 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.086 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.100 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.111 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.118 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.126 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41.134 UTC [11359] WARNING: there is no transaction in progress
2026-06-20 12:20:41,790 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 12:20:41,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 12:20:42,045 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 12:20:42,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:20:42,287 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 12:20:42,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:20:42,545 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 12:20:42,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:20:42,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:20:43,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9b3m.c.gcp-upstream.internal';"
2026-06-20 12:20:43,256 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 12:20:43,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 12:20:43,500 [I] DB_PATCHES have already been applied
I20260620 12:20:43.506413 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZkvxte/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 12:20:43.506502 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 12:20:43.510051 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34053
Jun 20, 2026 12:20:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:20:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:20:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:20:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:20:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:20:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:20:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 12:20:44.605785 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.389s user 0.006s sys 0.003s
Jun 20, 2026 12:21:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:21:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 12:21:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:21:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:21:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 12:21:05.941193 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 12:21:06 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958066, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 12:21:06.020687 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 12:21:06.029649 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 12:21:06.874866 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 12:21:06.874979 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms
I20260620 12:21:06.875138 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39427
2026-06-20 12:21:07,056 [I] DB FLAVOR :POSTGRES
2026-06-20 12:21:07,057 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:21:07,057 [I] Checking connection
2026-06-20 12:21:07,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 12:21:07,317 [I] connection success
2026-06-20 12:21:07,317 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 12:21:07,317 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 12:21:07,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 12:21:07,558 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 12:21:07,559 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:21:07,559 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 12:21:07,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39427/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 12:21:07,815 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 12:21:08.268106 392 mini_ranger_kms.cc:310] Created kms service
I20260620 12:21:08.438050 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 12:21:08.481141 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 12:21:08 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958066, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:21:08.850858 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 12:21:08.851049 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 12:21:08.851082 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 12:21:08.854647 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11539
I20260620 12:21:08.854787 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:40315
Jun 20, 2026 12:21:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:21:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:21:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:21:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:21:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:21:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 12:21:11.091856 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.071s user 0.000s sys 0.165s
I20260620 12:21:11.092008 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 12:21:11.092154 392 mini_ranger_kms.cc:401] 127.0.98.20:40315/kms/v1/keys
Jun 20 12:21:11 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 12:21:11 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:21:11 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:21:14 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:21:14.260216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43955
--ranger_config_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40315/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:21:14.370741 11590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:21:14.371044 11590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:21:14.371124 11590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:21:14.375191 11590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:21:14.375288 11590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:21:14.375330 11590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:21:14.375372 11590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:21:14.375411 11590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:21:14.380692 11590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:40315/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43955
--ranger_config_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:21:14.381784 11590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:21:14.382781 11590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:21:14.391618 11590 server_base.cc:1034] running on GCE node
W20260620 12:21:14.391516 11596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:21:14.391563 11595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:21:14.391530 11598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:21:14.392323 11590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:21:14.393452 11590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:21:14.394608 11590 hybrid_clock.cc:648] HybridClock initialized: now 1781958074394581 us; error 39 us; skew 500 ppm
Jun 20 12:21:14 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:21:14.397303 11590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:21:14.398159 11590 webserver.cc:460] Webserver started at http://127.0.98.62:44857/ using document root <none> and password file <none>
I20260620 12:21:14.398671 11590 fs_manager.cc:359] Metadata directory not provided
I20260620 12:21:14.398725 11590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:21:14.398924 11590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:21:14 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:21:14.443928 11590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e"
format_stamp: "Formatted at 2026-06-20 12:21:14 on dist-test-slave-9b3m"
server_key: "38ea8dc1d755ce6ce47d5b8f2cad2e8a"
server_key_iv: "f1807d9c6adde2ebb112254239afe634"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:14.444550 11590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e"
format_stamp: "Formatted at 2026-06-20 12:21:14 on dist-test-slave-9b3m"
server_key: "38ea8dc1d755ce6ce47d5b8f2cad2e8a"
server_key_iv: "f1807d9c6adde2ebb112254239afe634"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:14.448483 11590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 12:21:14.465906 11605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:14.467669 11590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:21:14.467809 11590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e"
format_stamp: "Formatted at 2026-06-20 12:21:14 on dist-test-slave-9b3m"
server_key: "38ea8dc1d755ce6ce47d5b8f2cad2e8a"
server_key_iv: "f1807d9c6adde2ebb112254239afe634"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:14.467943 11590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:21:14.504318 11590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:21:14.507608 11590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:21:14.507812 11590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:21:14.515799 11590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43955
I20260620 12:21:14.515861 11657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43955 every 8 connection(s)
I20260620 12:21:14.516801 11590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:21:14.517582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 11590
I20260620 12:21:14.517699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:21:14.517985 392 external_mini_cluster.cc:1428] Setting key 12c0a7ebfd7fe446ce5771a5068704a0
I20260620 12:21:14.520725 11658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:21:14.526760 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e: Bootstrap starting.
Jun 20 12:21:14 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:21:14.529384 11658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e: Neither blocks nor log segments found. Creating new log.
I20260620 12:21:14.530584 11658 log.cc:824] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:21:14.532615 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e: No bootstrap required, opened a new log
I20260620 12:21:14.534438 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:14.519581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.0.98.62:43955)
0620 12:21:14.520033 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:14.520052 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:14.520106 (+ 54us) server_negotiation.cc:407] Connection header received
0620 12:21:14.520975 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:14.520997 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:14.521458 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:14.521816 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:21:14.522658 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:14.524255 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:14.525485 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:14.526197 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:14.526369 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:14.529380 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:21:14.529386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:21:14.529402 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:21:14.529432 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:21:14.531392 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:14.531849 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:14.531855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:14.531860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:14.532030 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:14.532331 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:14.532334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:14.532336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:14.532788 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:21:14.532967 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:14.533261 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 12:21:14.533528 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260620 12:21:14.535894 11658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } }
I20260620 12:21:14.536152 11658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:21:14.536242 11658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fed6bfd8a7b48cabcaadc0bc9ce7c9e, State: Initialized, Role: FOLLOWER
I20260620 12:21:14.536688 11658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } }
I20260620 12:21:14.536870 11658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:21:14.536953 11658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:21:14.537092 11658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:21:14.537914 11658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } }
I20260620 12:21:14.538239 11658 leader_election.cc:302] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fed6bfd8a7b48cabcaadc0bc9ce7c9e; no voters:
I20260620 12:21:14.538542 11658 leader_election.cc:288] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:21:14.538607 11663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:21:14.538806 11663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [term 1 LEADER]: Becoming Leader. State: Replica: 4fed6bfd8a7b48cabcaadc0bc9ce7c9e, State: Running, Role: LEADER
I20260620 12:21:14.539132 11663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } }
I20260620 12:21:14.539726 11658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:21:14.540882 11664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } }
I20260620 12:21:14.541004 11664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [sys.catalog]: This master's current role is: LEADER
I20260620 12:21:14.541420 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fed6bfd8a7b48cabcaadc0bc9ce7c9e. Latest consensus state: current_term: 1 leader_uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fed6bfd8a7b48cabcaadc0bc9ce7c9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } }
I20260620 12:21:14.541574 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e [sys.catalog]: This master's current role is: LEADER
I20260620 12:21:14.541491 11672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:21:14.541967 11658 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:21:14.547760 11672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:21:14.558032 11672 catalog_manager.cc:1347] Generated new cluster ID: 787d681623d841bca6e9b206ed87de4d
I20260620 12:21:14.558101 11672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:21:14.578156 11672 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:21:14.579483 11672 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:21:14.586158 11672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e: Generated new TSK 0
I20260620 12:21:14.587018 11672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:21:15.240389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40315/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:21:15.364989 11703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:21:15.365316 11703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:21:15.365393 11703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:21:15.369468 11703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:21:15.369544 11703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:21:15.369642 11703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:21:15.375240 11703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:40315/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:21:15.376324 11703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:21:15.377401 11703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:21:15.388088 11709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:21:15.388214 11703 server_base.cc:1034] running on GCE node
W20260620 12:21:15.388090 11708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:21:15.388111 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:21:15.388936 11703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:21:15.389688 11703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:21:15.390874 11703 hybrid_clock.cc:648] HybridClock initialized: now 1781958075390847 us; error 46 us; skew 500 ppm
Jun 20 12:21:15 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:21:15.393849 11703 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:21:15.394822 11703 webserver.cc:460] Webserver started at http://127.0.98.1:42399/ using document root <none> and password file <none>
I20260620 12:21:15.395382 11703 fs_manager.cc:359] Metadata directory not provided
I20260620 12:21:15.395469 11703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:21:15.395712 11703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:21:15 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:21:15.411430 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "43ed240549e342ccaf130f5c5f150769"
format_stamp: "Formatted at 2026-06-20 12:21:15 on dist-test-slave-9b3m"
server_key: "653f904f33f0a43e4ddfd6d8884675ba"
server_key_iv: "701f98632433d84f09c98a5594f81231"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:15.412503 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "43ed240549e342ccaf130f5c5f150769"
format_stamp: "Formatted at 2026-06-20 12:21:15 on dist-test-slave-9b3m"
server_key: "653f904f33f0a43e4ddfd6d8884675ba"
server_key_iv: "701f98632433d84f09c98a5594f81231"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:15.421808 11703 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260620 12:21:15.447274 11720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:15.449162 11703 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:21:15.449296 11703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "43ed240549e342ccaf130f5c5f150769"
format_stamp: "Formatted at 2026-06-20 12:21:15 on dist-test-slave-9b3m"
server_key: "653f904f33f0a43e4ddfd6d8884675ba"
server_key_iv: "701f98632433d84f09c98a5594f81231"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:15.449831 11703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:21:15.471571 11703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:21:15.475044 11703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:21:15.475231 11703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:21:15.476359 11703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:21:15.477439 11703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:21:15.477505 11703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:15.477566 11703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:21:15.477602 11703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:15.488507 11703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34369
I20260620 12:21:15.488600 11833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34369 every 8 connection(s)
I20260620 12:21:15.489611 11703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:21:15.497941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 11703
I20260620 12:21:15.498075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:21:15.498371 392 external_mini_cluster.cc:1428] Setting key 4f15ba6519da8e1467f5fcf2a26c5f90
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 20 12:21:15 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:21:15.520105 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:15.491606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52793 (local address 127.0.98.62:43955)
0620 12:21:15.491890 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:15.491898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:15.492746 (+ 848us) server_negotiation.cc:407] Connection header received
0620 12:21:15.493966 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:15.493970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:15.494024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:15.494087 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:21:15.495624 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:15.496491 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:15.510964 (+ 14473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:15.511302 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:15.511398 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:15.515502 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:21:15.515506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:21:15.515509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:21:15.515537 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:21:15.517448 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:15.518147 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:15.518153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:15.518158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:15.518252 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:15.518677 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:15.518683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:15.518688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:15.518956 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:21:15.519043 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:15.519778 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 12:21:15.519923 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260620 12:21:15.521983 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:15.492058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:52793)
0620 12:21:15.492588 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:15.492643 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 12:21:15.493728 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:21:15.494490 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:21:15.494499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:21:15.494829 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:21:15.495456 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:15.495475 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:15.509120 (+ 13645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:15.509125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:15.510733 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:15.510750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:15.511640 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:15.511644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:15.511864 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:15.513226 (+ 1362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:21:15.513250 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:21:15.515273 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:21:15.517597 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:21:15.517603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:21:15.517618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:21:15.518027 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:21:15.518373 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:21:15.518377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:21:15.518379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:21:15.518566 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:21:15.519113 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:21:15.519122 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:21:15.519470 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 12:21:15.520577 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0620 12:21:15.520908 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260620 12:21:15.523612 11834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260620 12:21:15.523844 11834 heartbeater.cc:464] Registering TS with master...
I20260620 12:21:15.524538 11834 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260620 12:21:15.526507 11622 ts_manager.cc:194] Registered new tserver with Master: 43ed240549e342ccaf130f5c5f150769 (127.0.98.1:34369)
I20260620 12:21:15.528156 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:21:15.548784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40315/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:21:15 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 12:21:15.706755 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:21:15.707099 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:21:15.707159 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:21:15.713107 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:21:15.713227 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:21:15.713340 11842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:21:15.721114 11842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:40315/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:21:15.722445 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:21:15.723814 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:21:15.736866 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:21:15.737439 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:21:15.737254 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:21:15.738958 11842 server_base.cc:1034] running on GCE node
I20260620 12:21:15.739446 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:21:15.740195 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:21:15.741369 11842 hybrid_clock.cc:648] HybridClock initialized: now 1781958075741306 us; error 78 us; skew 500 ppm
Jun 20 12:21:15 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:21:15.745424 11842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:21:15.746662 11842 webserver.cc:460] Webserver started at http://127.0.98.2:36591/ using document root <none> and password file <none>
I20260620 12:21:15.747387 11842 fs_manager.cc:359] Metadata directory not provided
I20260620 12:21:15.747483 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:21:15.747711 11842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:21:15 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:21:15.766804 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "85559fc24e0045ce8f9cd58b8ca594ac"
format_stamp: "Formatted at 2026-06-20 12:21:15 on dist-test-slave-9b3m"
server_key: "aa721216715692c747b7c6698eca0b4a"
server_key_iv: "7b61f1a03bb0d5e456853e8dbabc3e89"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:15.768095 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "85559fc24e0045ce8f9cd58b8ca594ac"
format_stamp: "Formatted at 2026-06-20 12:21:15 on dist-test-slave-9b3m"
server_key: "aa721216715692c747b7c6698eca0b4a"
server_key_iv: "7b61f1a03bb0d5e456853e8dbabc3e89"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:15.773893 11842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:21:15.805267 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:15.807168 11842 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260620 12:21:15.807307 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "85559fc24e0045ce8f9cd58b8ca594ac"
format_stamp: "Formatted at 2026-06-20 12:21:15 on dist-test-slave-9b3m"
server_key: "aa721216715692c747b7c6698eca0b4a"
server_key_iv: "7b61f1a03bb0d5e456853e8dbabc3e89"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:15.807456 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:21:15.830940 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:21:15.834717 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:21:15.834983 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:21:15.835677 11842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:21:15.836781 11842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:21:15.836889 11842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:15.837013 11842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:21:15.837127 11842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:15.851565 11842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45047
I20260620 12:21:15.851634 11971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45047 every 8 connection(s)
I20260620 12:21:15.852870 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:21:15.862736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 11842
I20260620 12:21:15.862875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:21:15.863158 392 external_mini_cluster.cc:1428] Setting key 8058383c5b7cb8ed6d9dec43a4e02160
Jun 20 12:21:15 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:21:15.870479 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:15.854908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47551 (local address 127.0.98.62:43955)
0620 12:21:15.855033 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:15.855038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:15.856678 (+ 1640us) server_negotiation.cc:407] Connection header received
0620 12:21:15.857901 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:15.857905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:15.857960 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:15.858056 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:21:15.859417 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:15.860280 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:15.862058 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:15.862427 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:15.862510 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:15.865954 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:21:15.865958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:21:15.865960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:21:15.865990 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:21:15.867923 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:15.868571 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:15.868576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:15.868578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:15.868674 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:15.869131 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:15.869134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:15.869136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:15.869375 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:21:15.869459 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:15.870157 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 12:21:15.870291 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:21:15.872364 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:15.855424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.2:47551)
0620 12:21:15.856480 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:15.856538 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 12:21:15.857643 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:21:15.858248 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:21:15.858258 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:21:15.858607 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:21:15.859251 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:15.859272 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:15.860423 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:15.860427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:15.861917 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:15.861928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:15.862763 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:15.862768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:15.862983 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:15.863674 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:21:15.863695 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:21:15.865751 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:21:15.868057 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:21:15.868063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:21:15.868079 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:21:15.868423 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:21:15.868779 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:21:15.868782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:21:15.868784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:21:15.868986 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:21:15.869525 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:21:15.869532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:21:15.869874 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 12:21:15.870957 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0620 12:21:15.871291 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":158,"threads_started":1}
I20260620 12:21:15.875242 11972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260620 12:21:15.875504 11972 heartbeater.cc:464] Registering TS with master...
I20260620 12:21:15.876073 11972 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260620 12:21:15.877269 11622 ts_manager.cc:194] Registered new tserver with Master: 85559fc24e0045ce8f9cd58b8ca594ac (127.0.98.2:45047)
I20260620 12:21:15.877897 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47551
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:21:15.913170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40315/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:21:16.058782 11977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:21:16.059242 11977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:21:16.059350 11977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:21:16.066696 11977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:21:16.066869 11977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:21:16.067049 11977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:21:16.075814 11977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:40315/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:21:16.077520 11977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:21:16.078943 11977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 12:21:16 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 12:21:16.092342 11982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:21:16.092845 11977 server_base.cc:1034] running on GCE node
W20260620 12:21:16.093221 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:21:16.094789 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:21:16.095409 11977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:21:16.096156 11977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:21:16.097333 11977 hybrid_clock.cc:648] HybridClock initialized: now 1781958076097299 us; error 46 us; skew 500 ppm
Jun 20 12:21:16 dist-test-slave-9b3m krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:21:16.101131 11977 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:21:16.102211 11977 webserver.cc:460] Webserver started at http://127.0.98.3:46483/ using document root <none> and password file <none>
I20260620 12:21:16.102885 11977 fs_manager.cc:359] Metadata directory not provided
I20260620 12:21:16.102962 11977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:21:16.103209 11977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:21:16 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:21:16.119727 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "6b890745bc034440a503da458b73258a"
format_stamp: "Formatted at 2026-06-20 12:21:16 on dist-test-slave-9b3m"
server_key: "2913e1212098e8e2e5db90323c784615"
server_key_iv: "4088a486d5a3f872262810d85cf6b9a7"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:16.120472 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6b890745bc034440a503da458b73258a"
format_stamp: "Formatted at 2026-06-20 12:21:16 on dist-test-slave-9b3m"
server_key: "2913e1212098e8e2e5db90323c784615"
server_key_iv: "4088a486d5a3f872262810d85cf6b9a7"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:16.128189 11977 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260620 12:21:16.166712 11992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:16.170517 11977 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.002s sys 0.001s
I20260620 12:21:16.170681 11977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "6b890745bc034440a503da458b73258a"
format_stamp: "Formatted at 2026-06-20 12:21:16 on dist-test-slave-9b3m"
server_key: "2913e1212098e8e2e5db90323c784615"
server_key_iv: "4088a486d5a3f872262810d85cf6b9a7"
server_key_version: "kuduclusterkey@0"
I20260620 12:21:16.170846 11977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:21:16.192018 11977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:21:16.195583 11977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:21:16.195775 11977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:21:16.196398 11977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:21:16.197356 11977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:21:16.197413 11977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:16.197481 11977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:21:16.197511 11977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:21:16.207046 11977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43943
I20260620 12:21:16.208071 11977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:21:16.213533 12105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43943 every 8 connection(s)
I20260620 12:21:16.217211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 11977
I20260620 12:21:16.217347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:21:16.217657 392 external_mini_cluster.cc:1428] Setting key 0339cb0b0ab2c2c8cff1ba1816526c3f
Jun 20 12:21:16 dist-test-slave-9b3m krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:21:16.230229 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.217010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40631 (local address 127.0.98.62:43955)
0620 12:21:16.217186 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:16.217191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:16.217208 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:21:16.218746 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:16.218751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:16.218807 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:16.218865 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:21:16.220584 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.221389 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.223152 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.223488 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.223588 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.226310 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:21:16.226314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:21:16.226317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:21:16.226351 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:21:16.227965 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:16.228514 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:16.228520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:16.228522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:16.228606 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:16.228976 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:16.228981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:16.228983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:16.229251 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:21:16.229362 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:16.229910 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 12:21:16.230045 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 12:21:16.231742 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.215924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.3:40631)
0620 12:21:16.216515 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:16.216577 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:21:16.218462 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:21:16.219295 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:21:16.219305 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:21:16.219683 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:21:16.220414 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.220434 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.221514 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.221518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.223028 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.223040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.223646 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.223650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.223870 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.224620 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:21:16.224637 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:21:16.226130 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:21:16.228108 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:21:16.228115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:21:16.228130 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:21:16.228404 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:21:16.228716 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:21:16.228721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:21:16.228723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:21:16.228874 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:21:16.229371 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:21:16.229380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:21:16.229657 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 12:21:16.229867 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 12:21:16.230452 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":143,"threads_started":1}
I20260620 12:21:16.232905 12106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260620 12:21:16.233178 12106 heartbeater.cc:464] Registering TS with master...
I20260620 12:21:16.233701 12106 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260620 12:21:16.234793 11622 ts_manager.cc:194] Registered new tserver with Master: 6b890745bc034440a503da458b73258a (127.0.98.3:43943)
I20260620 12:21:16.235395 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40631
I20260620 12:21:16.243817 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:21:16.255789 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.246121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.0.98.62:43955)
0620 12:21:16.246272 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:16.246277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:16.246523 (+ 246us) server_negotiation.cc:407] Connection header received
0620 12:21:16.246647 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:16.246649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:16.246691 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:16.246763 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:21:16.247655 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.248218 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.249426 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.249691 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.249786 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.251649 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:21:16.251652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:21:16.251654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:21:16.251687 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:21:16.253414 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:16.253924 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:16.253929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:16.253933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:16.254032 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:21:16.254832 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:21:16.254835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:21:16.254836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:21:16.255022 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:21:16.255135 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:16.255483 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 12:21:16.255581 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:21:16.259179 11622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:21:16.262467 11622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:21:16.278445 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.270779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43943 (local address 127.0.0.1:38390)
0620 12:21:16.271744 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:16.271789 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 12:21:16.271914 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:21:16.273765 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:21:16.273768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:21:16.273786 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:21:16.273869 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.273874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.275321 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.275325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.276783 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.276793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.277861 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.277865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.278026 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.278070 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 12:21:16.278160 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:21:16.278296 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":26,"thread_start_us":90,"threads_started":1}
I20260620 12:21:16.279278 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.270950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34369 (local address 127.0.0.1:59868)
0620 12:21:16.271943 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:16.271957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:21:16.272027 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:21:16.273175 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:21:16.273181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:21:16.273219 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:21:16.273314 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.273319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.275641 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.275644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.276787 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.276793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.278903 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.278908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.278987 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.279007 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:21:16.279073 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 12:21:16.279137 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":21,"thread_start_us":97,"threads_started":1}
I20260620 12:21:16.279680 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.271971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.0.98.1:34369)
0620 12:21:16.272268 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:16.272277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:16.272301 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:21:16.272374 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:16.272380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:16.272624 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:16.272784 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:21:16.273428 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.274607 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.276915 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.278558 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.278666 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.278836 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:16.279452 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 12:21:16.279548 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
I20260620 12:21:16.280174 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.271560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.98.3:43943)
0620 12:21:16.272269 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:16.272279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:16.272300 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:21:16.272374 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:16.272380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:16.273613 (+ 1233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:16.273999 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:21:16.274170 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.275174 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.276973 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.277740 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.279282 (+ 1542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.279963 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:16.280003 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 12:21:16.280067 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":131,"threads_started":1}
I20260620 12:21:16.282720 12040 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0cb8ff1bcd43a5897a670075fbe99d (DEFAULT_TABLE table=test-table [id=e75774b8e46047bd983a602ca6f19c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:21:16.282725 11768 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0cb8ff1bcd43a5897a670075fbe99d (DEFAULT_TABLE table=test-table [id=e75774b8e46047bd983a602ca6f19c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:21:16.283800 11768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0cb8ff1bcd43a5897a670075fbe99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:21:16.283800 12040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0cb8ff1bcd43a5897a670075fbe99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:21:16.292882 12124 tablet_bootstrap.cc:492] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: Bootstrap starting.
I20260620 12:21:16.295766 12124 tablet_bootstrap.cc:654] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: Neither blocks nor log segments found. Creating new log.
I20260620 12:21:16.296727 12124 log.cc:824] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:21:16.297441 12125 tablet_bootstrap.cc:492] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769: Bootstrap starting.
I20260620 12:21:16.298084 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.271486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45047 (local address 127.0.0.1:37630)
0620 12:21:16.278613 (+ 7127us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:16.278628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:21:16.278694 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:21:16.292360 (+ 13666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:21:16.292363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:21:16.292383 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:21:16.292472 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.292479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.295204 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.295208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.296335 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.296345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.297696 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.297710 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.297796 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.297825 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 12:21:16.297876 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 12:21:16.297951 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7061,"thread_start_us":75,"threads_started":1}
I20260620 12:21:16.299678 12125 tablet_bootstrap.cc:654] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769: Neither blocks nor log segments found. Creating new log.
I20260620 12:21:16.300515 12125 log.cc:824] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769: Log is configured to *not* fsync() on all Append() calls
I20260620 12:21:16.300967 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.272272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.0.98.2:45047)
0620 12:21:16.273075 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:16.273087 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:16.279522 (+ 6435us) server_negotiation.cc:407] Connection header received
0620 12:21:16.279601 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:16.279607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:16.292171 (+ 12564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:16.292387 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:21:16.292606 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.295071 (+ 2465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.296588 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.297493 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.298461 (+ 968us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.298602 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:16.298652 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 12:21:16.298757 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":124,"threads_started":1}
I20260620 12:21:16.302726 12124 tablet_bootstrap.cc:492] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: No bootstrap required, opened a new log
I20260620 12:21:16.302731 12125 tablet_bootstrap.cc:492] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769: No bootstrap required, opened a new log
I20260620 12:21:16.302924 12124 ts_tablet_manager.cc:1397] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260620 12:21:16.302922 12125 ts_tablet_manager.cc:1397] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:21:16.307317 11905 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0cb8ff1bcd43a5897a670075fbe99d (DEFAULT_TABLE table=test-table [id=e75774b8e46047bd983a602ca6f19c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:21:16.307984 12124 raft_consensus.cc:348] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.308342 12124 raft_consensus.cc:374] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:21:16.308437 11905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0cb8ff1bcd43a5897a670075fbe99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:21:16.308434 12124 raft_consensus.cc:729] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b890745bc034440a503da458b73258a, State: Initialized, Role: FOLLOWER
I20260620 12:21:16.309026 12124 consensus_queue.cc:260] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.310168 12124 ts_tablet_manager.cc:1428] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260620 12:21:16.310563 12106 heartbeater.cc:503] Master 127.0.98.62:43955 was elected leader, sending a full tablet report...
I20260620 12:21:16.315145 12125 raft_consensus.cc:348] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.315425 12125 raft_consensus.cc:374] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:21:16.315490 12125 raft_consensus.cc:729] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ed240549e342ccaf130f5c5f150769, State: Initialized, Role: FOLLOWER
I20260620 12:21:16.318260 12129 tablet_bootstrap.cc:492] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Bootstrap starting.
I20260620 12:21:16.319093 12125 consensus_queue.cc:260] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.320129 12125 ts_tablet_manager.cc:1428] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260620 12:21:16.320480 11834 heartbeater.cc:503] Master 127.0.98.62:43955 was elected leader, sending a full tablet report...
I20260620 12:21:16.320789 12129 tablet_bootstrap.cc:654] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Neither blocks nor log segments found. Creating new log.
I20260620 12:21:16.322548 12129 log.cc:824] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Log is configured to *not* fsync() on all Append() calls
I20260620 12:21:16.326052 12129 tablet_bootstrap.cc:492] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: No bootstrap required, opened a new log
I20260620 12:21:16.326301 12129 ts_tablet_manager.cc:1397] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 12:21:16.329160 12129 raft_consensus.cc:348] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.329388 12129 raft_consensus.cc:374] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:21:16.329447 12129 raft_consensus.cc:729] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85559fc24e0045ce8f9cd58b8ca594ac, State: Initialized, Role: FOLLOWER
I20260620 12:21:16.329897 12129 consensus_queue.cc:260] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.330511 11972 heartbeater.cc:503] Master 127.0.98.62:43955 was elected leader, sending a full tablet report...
I20260620 12:21:16.331756 12129 ts_tablet_manager.cc:1428] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:21:16.351869 12128 raft_consensus.cc:482] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:21:16.352102 12128 raft_consensus.cc:504] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.353340 12128 leader_election.cc:288] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43ed240549e342ccaf130f5c5f150769 (127.0.98.1:34369), 85559fc24e0045ce8f9cd58b8ca594ac (127.0.98.2:45047)
W20260620 12:21:16.355036 11973 tablet.cc:2369] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:21:16.358465 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.353584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34369 (local address 127.0.98.3:37007)
0620 12:21:16.353729 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:16.353747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:21:16.353849 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:21:16.354202 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:21:16.354206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:21:16.354225 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:21:16.354338 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.354344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.355384 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.355387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.356527 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.356540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.358066 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.358069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.358165 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.358187 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:21:16.358254 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 12:21:16.358322 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 12:21:16.358929 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.353732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37007 (local address 127.0.98.1:34369)
0620 12:21:16.353872 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:16.353877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:16.353895 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:21:16.353952 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:16.353955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:16.353999 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:16.354085 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:21:16.354462 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.355251 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.356712 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.357844 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.357947 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.358011 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:16.358732 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 12:21:16.358817 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:21:16.359452 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.354234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47245 (local address 127.0.98.2:45047)
0620 12:21:16.354463 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:16.354470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:16.354485 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:21:16.354642 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:16.354645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:16.354691 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:16.354736 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:21:16.355944 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.356665 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.358137 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.358738 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.358816 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.358886 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:16.359242 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 12:21:16.359319 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 12:21:16.359484 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0cb8ff1bcd43a5897a670075fbe99d" candidate_uuid: "6b890745bc034440a503da458b73258a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ed240549e342ccaf130f5c5f150769" is_pre_election: true
I20260620 12:21:16.359802 11783 raft_consensus.cc:2454] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b890745bc034440a503da458b73258a in term 0.
I20260620 12:21:16.359889 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.353849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45047 (local address 127.0.98.3:47245)
0620 12:21:16.354423 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:16.354436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:21:16.354523 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:21:16.355649 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:21:16.355652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:21:16.355670 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:21:16.355812 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.355818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.356819 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.356822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.357991 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:21:16.358004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.358999 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.359004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:21:16.359108 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.359128 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:21:16.359645 (+ 517us) client_negotiation.cc:240] Negotiation successful
0620 12:21:16.359721 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":18,"thread_start_us":128,"threads_started":1}
I20260620 12:21:16.360524 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0cb8ff1bcd43a5897a670075fbe99d" candidate_uuid: "6b890745bc034440a503da458b73258a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" is_pre_election: true
I20260620 12:21:16.360790 11910 raft_consensus.cc:2454] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b890745bc034440a503da458b73258a in term 0.
I20260620 12:21:16.361326 11993 leader_election.cc:302] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b890745bc034440a503da458b73258a, 85559fc24e0045ce8f9cd58b8ca594ac; no voters:
I20260620 12:21:16.361858 12128 raft_consensus.cc:2781] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:21:16.361959 12128 raft_consensus.cc:482] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:21:16.362015 12128 raft_consensus.cc:3037] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:21:16.363396 12128 raft_consensus.cc:504] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.363775 12128 leader_election.cc:288] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [CANDIDATE]: Term 1 election: Requested vote from peers 43ed240549e342ccaf130f5c5f150769 (127.0.98.1:34369), 85559fc24e0045ce8f9cd58b8ca594ac (127.0.98.2:45047)
I20260620 12:21:16.364259 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0cb8ff1bcd43a5897a670075fbe99d" candidate_uuid: "6b890745bc034440a503da458b73258a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ed240549e342ccaf130f5c5f150769"
I20260620 12:21:16.364373 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0cb8ff1bcd43a5897a670075fbe99d" candidate_uuid: "6b890745bc034440a503da458b73258a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85559fc24e0045ce8f9cd58b8ca594ac"
I20260620 12:21:16.364388 11783 raft_consensus.cc:3037] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:21:16.364502 11910 raft_consensus.cc:3037] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:21:16.365664 11783 raft_consensus.cc:2454] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b890745bc034440a503da458b73258a in term 1.
I20260620 12:21:16.365815 11910 raft_consensus.cc:2454] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b890745bc034440a503da458b73258a in term 1.
I20260620 12:21:16.365971 11993 leader_election.cc:302] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ed240549e342ccaf130f5c5f150769, 6b890745bc034440a503da458b73258a; no voters:
I20260620 12:21:16.366134 12128 raft_consensus.cc:2781] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:21:16.366384 12128 raft_consensus.cc:686] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 1 LEADER]: Becoming Leader. State: Replica: 6b890745bc034440a503da458b73258a, State: Running, Role: LEADER
I20260620 12:21:16.366740 12128 consensus_queue.cc:237] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } }
I20260620 12:21:16.370352 11621 catalog_manager.cc:5495] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b890745bc034440a503da458b73258a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6b890745bc034440a503da458b73258a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b890745bc034440a503da458b73258a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 } health_report { overall_health: UNKNOWN } } }
I20260620 12:21:16.382622 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.378955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38400 (local address 127.0.98.3:43943)
0620 12:21:16.379071 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 12:21:16.379075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:21:16.379136 (+ 61us) server_negotiation.cc:407] Connection header received
0620 12:21:16.379333 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:21:16.379341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:21:16.379395 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:21:16.379510 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:21:16.379741 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.380232 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.381164 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:21:16.381388 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:21:16.381500 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:21:16.381801 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:21:16.382202 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:21:16.382332 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 12:21:16.382411 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 12:21:16.382463 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:21:16.388226 11910 raft_consensus.cc:1261] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 1 FOLLOWER]: Refusing update from remote peer 6b890745bc034440a503da458b73258a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:21:16.388226 11783 raft_consensus.cc:1261] T 2a0cb8ff1bcd43a5897a670075fbe99d P 43ed240549e342ccaf130f5c5f150769 [term 1 FOLLOWER]: Refusing update from remote peer 6b890745bc034440a503da458b73258a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:21:16.388931 12128 consensus_queue.cc:1035] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85559fc24e0045ce8f9cd58b8ca594ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:21:16.389261 12134 consensus_queue.cc:1035] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ed240549e342ccaf130f5c5f150769" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:21:16.396610 12137 mvcc.cc:205] Tried to move back new op lower bound from 7298900280879345664 to 7298900280801452032. Current Snapshot: MvccSnapshot[applied={T|T < 7298900280879345664}]
I20260620 12:21:16.399585 12139 mvcc.cc:205] Tried to move back new op lower bound from 7298900280879345664 to 7298900280801452032. Current Snapshot: MvccSnapshot[applied={T|T < 7298900280879345664}]
I20260620 12:21:16.414184 11621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:21:16.414379 11621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:21:16.417035 11621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fed6bfd8a7b48cabcaadc0bc9ce7c9e: Sending DeleteTablet for 3 replicas of tablet 2a0cb8ff1bcd43a5897a670075fbe99d
I20260620 12:21:16.417886 12040 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0cb8ff1bcd43a5897a670075fbe99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:21:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38390
I20260620 12:21:16.417865 11768 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0cb8ff1bcd43a5897a670075fbe99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:21:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260620 12:21:16.419257 11905 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0cb8ff1bcd43a5897a670075fbe99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:21:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37630
I20260620 12:21:16.420414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 11703
I20260620 12:21:16.426731 12147 tablet_replica.cc:330] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: stopping tablet replica
I20260620 12:21:16.427098 12147 raft_consensus.cc:2229] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:21:16.427502 12147 raft_consensus.cc:2258] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:21:16.427882 12149 tablet_replica.cc:330] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: stopping tablet replica
I20260620 12:21:16.428176 12149 raft_consensus.cc:2229] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:21:16.428507 12149 raft_consensus.cc:2258] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:21:16.430186 12147 ts_tablet_manager.cc:1905] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:21:16.433193 12147 ts_tablet_manager.cc:1918] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:21:16.433295 12147 log.cc:1196] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/2a0cb8ff1bcd43a5897a670075fbe99d
I20260620 12:21:16.433629 12147 ts_tablet_manager.cc:1939] T 2a0cb8ff1bcd43a5897a670075fbe99d P 6b890745bc034440a503da458b73258a: Deleting consensus metadata
I20260620 12:21:16.434839 11609 catalog_manager.cc:4841] TS 6b890745bc034440a503da458b73258a (127.0.98.3:43943): tablet 2a0cb8ff1bcd43a5897a670075fbe99d (table test-table [id=e75774b8e46047bd983a602ca6f19c0d]) successfully deleted
I20260620 12:21:16.436422 12149 ts_tablet_manager.cc:1905] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:21:16.438879 12149 ts_tablet_manager.cc:1918] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:21:16.438975 12149 log.cc:1196] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/2a0cb8ff1bcd43a5897a670075fbe99d
I20260620 12:21:16.439273 12149 ts_tablet_manager.cc:1939] T 2a0cb8ff1bcd43a5897a670075fbe99d P 85559fc24e0045ce8f9cd58b8ca594ac: Deleting consensus metadata
W20260620 12:21:16.440150 11606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34369 (error 108)
I20260620 12:21:16.441159 11606 catalog_manager.cc:4841] TS 85559fc24e0045ce8f9cd58b8ca594ac (127.0.98.2:45047): tablet 2a0cb8ff1bcd43a5897a670075fbe99d (table test-table [id=e75774b8e46047bd983a602ca6f19c0d]) successfully deleted
I20260620 12:21:16.441228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 11842
I20260620 12:21:16.445001 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:21:16.444619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34369 (local address 127.0.0.1:59878)
0620 12:21:16.444806 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 12:21:16.444911 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 12:21:16.445220 11606 catalog_manager.cc:4584] TS 43ed240549e342ccaf130f5c5f150769 (127.0.98.1:34369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a0cb8ff1bcd43a5897a670075fbe99d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34369: connect: Connection refused (error 111)
I20260620 12:21:16.451697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 11977
I20260620 12:21:16.459688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 11590
I20260620 12:21:16.466637 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 12:21:16.801859 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 12:21:16.801998 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 12:21:17.147800 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 12:21:17.148 UTC [11176] LOG: received smart shutdown request
2026-06-20 12:21:17.151 UTC [11176] LOG: background worker "logical replication launcher" (PID 11185) exited with exit code 1
2026-06-20 12:21:17.151 UTC [11180] LOG: shutting down
2026-06-20 12:21:17.304 UTC [11176] LOG: database system is shut down
2026-06-20T12:21:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45819 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:21:17 dist-test-slave-9b3m krb5kdc[12160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:21:17 dist-test-slave-9b3m krb5kdc[12160](info): set up 2 sockets
Jun 20 12:21:17 dist-test-slave-9b3m krb5kdc[12160](info): commencing operation
krb5kdc: starting...
W20260620 12:21:18.513695 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:21:18 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:21:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:21:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 12:21:18.683902 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZkvxte/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/postgres -l logfile start
2026-06-20 12:21:21.199 UTC [12190] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 12:21:21.199 UTC [12190] LOG: listening on IPv4 address "127.0.98.20", port 45383
2026-06-20 12:21:21.203 UTC [12190] LOG: listening on Unix socket "/tmp/.s.PGSQL.45383"
2026-06-20 12:21:21.225 UTC [12193] LOG: database system was shut down at 2026-06-20 12:21:19 UTC
2026-06-20 12:21:21.228 UTC [12190] LOG: database system is ready to accept connections
I20260620 12:21:22.185420 392 mini_postgres.cc:96] Postgres bound to 45383
2026-06-20 12:21:22.191 UTC [12201] FATAL: database "slave" does not exist
127.0.98.20:45383 - accepting connections
I20260620 12:21:22.192389 392 mini_ranger.cc:161] Starting Ranger...
I20260620 12:21:22.209496 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 12:21:22.693991 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 12:21:22.694078 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-admin
2026-06-20 12:21:22,972 [I] DB FLAVOR :POSTGRES
2026-06-20 12:21:22,972 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:21:22,972 [I] Checking connection..
2026-06-20 12:21:22,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 12:21:23,212 [I] Checking connection passed.
2026-06-20 12:21:23,212 [I] --------- Verifying version history table ---------
2026-06-20 12:21:23,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:21:23,471 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:21:23,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:21:23,731 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 12:21:23,731 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 12:21:23,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 12:21:23.983 UTC [12265] WARNING: there is no transaction in progress
2026-06-20 12:21:23,989 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 12:21:23,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 12:21:24,240 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 12:21:24,241 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:21:24,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 12:21:24,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 12:21:24,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9b3m.c.gcp-upstream.internal','N') ;"
2026-06-20 12:21:24,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:21:25,219 [I] Table x_portal_user does not exist in database ranger
2026-06-20 12:21:25,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:21:25,457 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 12:21:25,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:21:25,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:21:25,926 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 12:21:25,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 12:21:26.917 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:26.934 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:26.952 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:26.968 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:26.984 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.001 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.057 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.064 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.079 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.090 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.100 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.113 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.122 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.129 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.136 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27.143 UTC [12373] WARNING: there is no transaction in progress
2026-06-20 12:21:27,804 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 12:21:27,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 12:21:28,028 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 12:21:28,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 12:21:28,262 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 12:21:28,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 12:21:28,519 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 12:21:28,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 12:21:28,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 12:21:29,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9b3m.c.gcp-upstream.internal';"
2026-06-20 12:21:29,283 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 12:21:29,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 12:21:29,535 [I] DB_PATCHES have already been applied
I20260620 12:21:29.540891 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZkvxte/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 12:21:29.540946 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 12:21:29.544456 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:45989
Jun 20, 2026 12:21:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:21:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:21:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:21:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:21:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:21:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:21:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 12:21:30.676131 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.484s user 0.001s sys 0.009s
Jun 20, 2026 12:21:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:21:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 12:21:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:21:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:21:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 12:21:52.396466 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 12:21:52 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958112, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 12:21:52.497545 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 12:21:52.507941 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 12:21:53.466778 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 12:21:53.466864 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms
I20260620 12:21:53.466887 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45383
2026-06-20 12:21:53,636 [I] DB FLAVOR :POSTGRES
2026-06-20 12:21:53,636 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 12:21:53,636 [I] Checking connection
2026-06-20 12:21:53,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 12:21:53,893 [I] connection success
2026-06-20 12:21:53,893 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 12:21:53,893 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 12:21:53,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 12:21:54,141 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 12:21:54,141 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 12:21:54,141 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 12:21:54,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 12:21:54,405 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 12:21:54.881399 392 mini_ranger_kms.cc:310] Created kms service
I20260620 12:21:55.045176 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 12:21:55.088021 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 12:21:55 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958112, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:21:55.494836 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 12:21:55.495101 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZkvxte/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZkvxte/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 12:21:55.495129 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 12:21:55.499293 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12553
I20260620 12:21:55.499392 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36937
Jun 20, 2026 12:21:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:21:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:21:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:21:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:21:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:21:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 12:21:57 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:21:57.809113 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.312s user 0.013s sys 0.140s
I20260620 12:21:57.809229 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 12:21:57.809299 392 mini_ranger_kms.cc:401] 127.0.98.20:36937/kms/v1/keys
Jun 20 12:21:58 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:21:58 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:22:00 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:22:00.828678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41653
--ranger_config_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36937/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:00.942109 12604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:00.942373 12604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:00.942415 12604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:00.946623 12604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:00.946686 12604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:00.946707 12604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:00.946727 12604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:00.946777 12604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:00.952191 12604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36937/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41653
--ranger_config_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:00.953284 12604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:00.954371 12604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:00.963451 12609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:00.963466 12610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:00.963442 12612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:00.963646 12604 server_base.cc:1034] running on GCE node
I20260620 12:22:00.964632 12604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:00.966044 12604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:00.967264 12604 hybrid_clock.cc:648] HybridClock initialized: now 1781958120967213 us; error 51 us; skew 500 ppm
Jun 20 12:22:00 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:00.976626 12604 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:00.977663 12604 webserver.cc:460] Webserver started at http://127.0.98.62:39811/ using document root <none> and password file <none>
I20260620 12:22:00.978278 12604 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:00.978333 12604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:00.978559 12604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:22:00 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:01.034282 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "83536ce5e4274b53be4d7640b579f36a"
format_stamp: "Formatted at 2026-06-20 12:22:01 on dist-test-slave-9b3m"
server_key: "41035f71bf35a8a3cb891fa7c499d513"
server_key_iv: "a14d19ae136e58a60d506b5ff49d2727"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:01.035044 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "83536ce5e4274b53be4d7640b579f36a"
format_stamp: "Formatted at 2026-06-20 12:22:01 on dist-test-slave-9b3m"
server_key: "41035f71bf35a8a3cb891fa7c499d513"
server_key_iv: "a14d19ae136e58a60d506b5ff49d2727"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:01.040030 12604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 12:22:01.056555 12619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:01.058153 12604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 12:22:01.058286 12604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "83536ce5e4274b53be4d7640b579f36a"
format_stamp: "Formatted at 2026-06-20 12:22:01 on dist-test-slave-9b3m"
server_key: "41035f71bf35a8a3cb891fa7c499d513"
server_key_iv: "a14d19ae136e58a60d506b5ff49d2727"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:01.058426 12604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:01.076027 12604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:01.079521 12604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:01.079728 12604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:01.087379 12604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41653
I20260620 12:22:01.087406 12671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41653 every 8 connection(s)
I20260620 12:22:01.088326 12604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:01.091645 12672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:01.096431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 12604
I20260620 12:22:01.096571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:22:01.096858 392 external_mini_cluster.cc:1428] Setting key 6b29755b951f8289e1a3358deeb3ff39
I20260620 12:22:01.098627 12672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Bootstrap starting.
I20260620 12:22:01.101385 12672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:01.102221 12672 log.cc:824] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:01.104223 12672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: No bootstrap required, opened a new log
Jun 20 12:22:01 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:01.108419 12672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260620 12:22:01.108664 12672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:01.108737 12672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83536ce5e4274b53be4d7640b579f36a, State: Initialized, Role: FOLLOWER
I20260620 12:22:01.109258 12672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260620 12:22:01.109436 12672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:22:01.109544 12672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:22:01.109658 12672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:01.110594 12672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260620 12:22:01.111016 12672 leader_election.cc:302] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83536ce5e4274b53be4d7640b579f36a; no voters:
I20260620 12:22:01.111521 12672 leader_election.cc:288] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:22:01.111583 12677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:01.111806 12677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 1 LEADER]: Becoming Leader. State: Replica: 83536ce5e4274b53be4d7640b579f36a, State: Running, Role: LEADER
I20260620 12:22:01.112197 12677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260620 12:22:01.112362 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:01.098331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.0.98.62:41653)
0620 12:22:01.098721 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:01.098740 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:01.098783 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:22:01.099394 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:01.099410 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:01.099721 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:01.100049 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:01.100806 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:01.101945 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:01.102994 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:01.103701 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:01.103862 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:01.107013 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:01.107020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:01.107037 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:01.107066 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:01.109010 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:01.109556 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:01.109563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:01.109570 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:01.109702 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:01.110048 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:01.110051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:01.110053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:01.110547 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:01.110796 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:01.111038 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 12:22:01.111341 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260620 12:22:01.112794 12672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:01.113618 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83536ce5e4274b53be4d7640b579f36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260620 12:22:01.113782 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:01.114012 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83536ce5e4274b53be4d7640b579f36a. Latest consensus state: current_term: 1 leader_uuid: "83536ce5e4274b53be4d7640b579f36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260620 12:22:01.114133 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:01.114862 12686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:01.115134 12672 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:22:01.120970 12686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:01.131296 12686 catalog_manager.cc:1347] Generated new cluster ID: e6244917b3c14c6eb0a5822575988cec
I20260620 12:22:01.131361 12686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:22:01.151960 12686 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:22:01.153359 12686 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:22:01.163165 12686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Generated new TSK 0
I20260620 12:22:01.164268 12686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:22:01.925570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:02.074775 12717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:02.075060 12717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:02.075186 12717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:02.081158 12717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:02.081278 12717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:02.081410 12717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:22:02.089113 12717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:02.090520 12717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:02.091812 12717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:02.101809 12727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:02.101948 12725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:02.102185 12724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:02.102210 12717 server_base.cc:1034] running on GCE node
I20260620 12:22:02.102599 12717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:02.103221 12717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:02.104417 12717 hybrid_clock.cc:648] HybridClock initialized: now 1781958122104393 us; error 43 us; skew 500 ppm
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:02.107257 12717 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:22:02.108193 12717 webserver.cc:460] Webserver started at http://127.0.98.1:36107/ using document root <none> and password file <none>
I20260620 12:22:02.108851 12717 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:02.108938 12717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:02.109179 12717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:02.124069 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "cdcbdaf09cdf08df685864496b3a614c"
server_key_iv: "1ce879b6d3bb0a5aa0b95b1b7cb3c9d0"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.125169 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "cdcbdaf09cdf08df685864496b3a614c"
server_key_iv: "1ce879b6d3bb0a5aa0b95b1b7cb3c9d0"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.130463 12717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 12:22:02.150115 12734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:02.151901 12717 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260620 12:22:02.152101 12717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "cdcbdaf09cdf08df685864496b3a614c"
server_key_iv: "1ce879b6d3bb0a5aa0b95b1b7cb3c9d0"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.152321 12717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:02.174085 12717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:02.177829 12717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:02.178004 12717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:02.178653 12717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:02.179702 12717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:02.179761 12717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:02.179826 12717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:02.179900 12717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 12:22:02.190765 12717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36785
I20260620 12:22:02.190852 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36785 every 8 connection(s)
I20260620 12:22:02.191980 12717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:22:02.196384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 12717
I20260620 12:22:02.196493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:22:02.196802 392 external_mini_cluster.cc:1428] Setting key e7e1f0dab6f522f542724e6341104b66
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:02.210256 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:02.194553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.1:43711)
0620 12:22:02.195134 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:02.195189 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:02.196220 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:02.197102 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:02.197112 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:02.197470 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:02.198123 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:02.198143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.199430 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.199433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:02.200822 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:02.200833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.201565 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.201569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:02.201788 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:02.202502 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:02.202523 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:02.204336 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:02.206705 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:02.206713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:02.206733 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:02.207106 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:02.207476 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:02.207482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:02.207486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:02.207677 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:02.208195 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:02.208203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:02.208550 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 12:22:02.208767 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 12:22:02.209128 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":154,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:22:02.210978 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:02.195638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43711 (local address 127.0.98.62:41653)
0620 12:22:02.196645 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:02.196650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:02.196665 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:22:02.196709 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:02.196712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:02.196760 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:02.198319 (+ 1559us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:02.198429 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.199267 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.200991 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.201398 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.202767 (+ 1369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:02.204523 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:02.204529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:02.204535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:02.204565 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:02.206563 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:02.207233 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:02.207238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:02.207243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:02.207357 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:02.207798 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:02.207803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:02.207808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:02.208042 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:02.208136 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:02.210725 (+ 2589us) server_negotiation.cc:299] Negotiation successful
0620 12:22:02.210861 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":81,"threads_started":1}
I20260620 12:22:02.212531 12850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260620 12:22:02.212791 12850 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:22:02.213457 12850 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:02.216236 12636 ts_manager.cc:194] Registered new tserver with Master: b0e6d638cd1e48f4bb63ec8d9001e6d8 (127.0.98.1:36785)
I20260620 12:22:02.217410 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:22:02.232079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 12:22:02.358193 12856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:02.358469 12856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:02.358563 12856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:02.362990 12856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:02.363075 12856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:02.363176 12856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:22:02.368811 12856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:02.369984 12856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:02.371315 12856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:02.381742 12865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:02.382076 12863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:02.382186 12862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:02.382400 12856 server_base.cc:1034] running on GCE node
I20260620 12:22:02.382826 12856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:02.383445 12856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:02.386776 12856 hybrid_clock.cc:648] HybridClock initialized: now 1781958122386737 us; error 47 us; skew 500 ppm
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:02.389882 12856 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:22:02.391120 12856 webserver.cc:460] Webserver started at http://127.0.98.2:45449/ using document root <none> and password file <none>
I20260620 12:22:02.391710 12856 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:02.391808 12856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:02.392040 12856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:02.407719 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "02ce9577c3c24e14bf42ac691edee9ec"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "bd3e9c31c4e5075c79a72d51d13b2598"
server_key_iv: "a727e29e9e77234ae6023f94dc16a100"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.409473 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "02ce9577c3c24e14bf42ac691edee9ec"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "bd3e9c31c4e5075c79a72d51d13b2598"
server_key_iv: "a727e29e9e77234ae6023f94dc16a100"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.413805 12856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 12:22:02.432108 12872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:02.433917 12856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 12:22:02.434044 12856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "02ce9577c3c24e14bf42ac691edee9ec"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "bd3e9c31c4e5075c79a72d51d13b2598"
server_key_iv: "a727e29e9e77234ae6023f94dc16a100"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.434176 12856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:02.452237 12856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:02.455300 12856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:02.455525 12856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:02.456147 12856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:02.457036 12856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:02.457151 12856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:02.457242 12856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:02.457291 12856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:02.470042 12856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36869
I20260620 12:22:02.470996 12856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:22:02.472167 12985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36869 every 8 connection(s)
I20260620 12:22:02.474215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 12856
I20260620 12:22:02.474327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:22:02.474628 392 external_mini_cluster.cc:1428] Setting key 9714b61beecf2d76538d077bfb110fb2
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:02.493155 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:02.476220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47473 (local address 127.0.98.62:41653)
0620 12:22:02.476358 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:02.476363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:02.478541 (+ 2178us) server_negotiation.cc:407] Connection header received
0620 12:22:02.479855 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:02.479859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:02.479909 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:02.479961 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:02.482983 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.483832 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.485545 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.485891 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.485968 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:02.488985 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:02.488988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:02.488991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:02.489019 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:02.490773 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:02.491386 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:02.491389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:02.491391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:02.491474 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:02.491853 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:02.491856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:02.491857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:02.492064 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:02.492142 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:02.492812 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 12:22:02.492942 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:22:02.494915 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:02.475161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.2:47473)
0620 12:22:02.478384 (+ 3223us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:02.478443 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:02.479603 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:02.480383 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:02.480392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:02.480754 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:02.481473 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:02.481491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.483992 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.483995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:02.485408 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:02.485418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.486178 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.486182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:02.486369 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:02.487062 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:02.487082 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:02.488797 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:02.490911 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:02.490918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:02.490934 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:02.491269 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:02.491577 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:02.491580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:02.491582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:02.491747 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:02.492221 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:02.492228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:02.492540 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 12:22:02.493602 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0620 12:22:02.493921 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2987,"thread_start_us":160,"threads_started":1}
I20260620 12:22:02.496356 12986 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260620 12:22:02.496595 12986 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:02.497181 12986 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:02.498330 12636 ts_manager.cc:194] Registered new tserver with Master: 02ce9577c3c24e14bf42ac691edee9ec (127.0.98.2:36869)
I20260620 12:22:02.499092 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:22:02.536548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:02.707939 12991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:02.708287 12991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:02.708353 12991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:02.714391 12991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:02.714466 12991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:02.714553 12991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:22:02.722338 12991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:02.723735 12991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:02.725085 12991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:02.738615 12999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:02.739245 12996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:02.738694 12997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:02.740226 12991 server_base.cc:1034] running on GCE node
I20260620 12:22:02.740716 12991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:02.741618 12991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:02.742815 12991 hybrid_clock.cc:648] HybridClock initialized: now 1781958122742745 us; error 84 us; skew 500 ppm
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:02.746551 12991 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:22:02.747704 12991 webserver.cc:460] Webserver started at http://127.0.98.3:34133/ using document root <none> and password file <none>
I20260620 12:22:02.748457 12991 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:02.748596 12991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:02.748910 12991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:02.770032 12991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a260735b17f4c89a2f348c6bd1a7ba1"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "d7142e77b3e99114ee3460508e91c468"
server_key_iv: "ec0eefe154e92cac796a95acdb4fd4e4"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.770841 12991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a260735b17f4c89a2f348c6bd1a7ba1"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "d7142e77b3e99114ee3460508e91c468"
server_key_iv: "ec0eefe154e92cac796a95acdb4fd4e4"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.783878 12991 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.004s
I20260620 12:22:02.802101 13006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:02.803819 12991 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 12:22:02.803951 12991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "0a260735b17f4c89a2f348c6bd1a7ba1"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "d7142e77b3e99114ee3460508e91c468"
server_key_iv: "ec0eefe154e92cac796a95acdb4fd4e4"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:02.804085 12991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:02.828056 12991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:02.831178 12991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:02.831393 12991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:02.831962 12991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:02.832800 12991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:02.832881 12991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:02.832983 12991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:02.833027 12991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:02.843073 12991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44153
I20260620 12:22:02.843211 13119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44153 every 8 connection(s)
I20260620 12:22:02.844014 12991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:22:02.844734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 12991
I20260620 12:22:02.844854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:22:02.845177 392 external_mini_cluster.cc:1428] Setting key fd3e045d99c3bb3ec41e4a7aa4bbee42
Jun 20 12:22:02 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:02.865912 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:02.846835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52747 (local address 127.0.98.62:41653)
0620 12:22:02.846972 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:02.846976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:02.848290 (+ 1314us) server_negotiation.cc:407] Connection header received
0620 12:22:02.849321 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:02.849325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:02.849378 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:02.849453 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:02.850521 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.851092 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.852300 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.852519 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.852636 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:02.855613 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:02.855618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:02.855622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:02.855654 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:02.863250 (+ 7596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:02.863925 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:02.863930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:02.863934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:02.864028 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:02.864700 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:02.864703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:02.864705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:02.864966 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:02.865091 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:02.865633 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 12:22:02.865756 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:22:02.866936 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:02.847353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.3:52747)
0620 12:22:02.848155 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:02.848207 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:02.849118 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:02.849646 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:02.849653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:02.849919 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:02.850380 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:02.850392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.851237 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.851241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:02.852155 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:02.852166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:02.852668 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:02.852672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:02.852851 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:02.853446 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:02.853461 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:02.855219 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:02.863398 (+ 8179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:02.863407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:02.863426 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:02.863801 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:02.864386 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:02.864389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:02.864391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:02.864589 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:02.865105 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:02.865112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:02.865392 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 12:22:02.865593 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 12:22:02.865919 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":133,"threads_started":1}
I20260620 12:22:02.868012 13120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260620 12:22:02.868265 13120 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:02.868772 13120 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:02.869841 12636 ts_manager.cc:194] Registered new tserver with Master: 0a260735b17f4c89a2f348c6bd1a7ba1 (127.0.98.3:44153)
I20260620 12:22:02.870569 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52747
I20260620 12:22:02.871620 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:22:02.872337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 12717
I20260620 12:22:02.879789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 12856
I20260620 12:22:02.886669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 12991
I20260620 12:22:02.892949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 12604
I20260620 12:22:02.900936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--webserver_interface=127.0.98.62
--webserver_port=39811
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41653
--ranger_config_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36937/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36937/kms with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:03.008800 13123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:03.009099 13123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:03.009150 13123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:03.013137 13123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:03.013195 13123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:03.013219 13123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:03.013236 13123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:03.013252 13123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:03.018886 13123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36937/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41653
--ranger_config_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:03.019909 13123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:03.020944 13123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:03.029318 13129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:03.029274 13128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:03.029280 13131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:03.029698 13123 server_base.cc:1034] running on GCE node
I20260620 12:22:03.030174 13123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:03.031220 13123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:03.032387 13123 hybrid_clock.cc:648] HybridClock initialized: now 1781958123032364 us; error 33 us; skew 500 ppm
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:03.035233 13123 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:03.036058 13123 webserver.cc:460] Webserver started at http://127.0.98.62:39811/ using document root <none> and password file <none>
I20260620 12:22:03.036638 13123 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:03.036721 13123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:03.081540 13123 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:22:03.084251 13139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.085925 13123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:22:03.086072 13123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "83536ce5e4274b53be4d7640b579f36a"
format_stamp: "Formatted at 2026-06-20 12:22:01 on dist-test-slave-9b3m"
server_key: "41035f71bf35a8a3cb891fa7c499d513"
server_key_iv: "a14d19ae136e58a60d506b5ff49d2727"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:03.086644 13123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:03.102608 13123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:03.105257 13123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:03.105441 13123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:03.112252 13123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41653
I20260620 12:22:03.112287 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41653 every 8 connection(s)
I20260620 12:22:03.113333 13123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:03.117488 13192 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:22:03.117794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 13123
I20260620 12:22:03.118238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36785
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41653
--webserver_port=36107
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36937/kms with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:03.119202 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Bootstrap starting.
I20260620 12:22:03.135553 13192 log.cc:824] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:03.141997 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:22:03.142369 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Bootstrap complete.
I20260620 12:22:03.147049 13192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260620 12:22:03.147418 13192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83536ce5e4274b53be4d7640b579f36a, State: Initialized, Role: FOLLOWER
I20260620 12:22:03.147897 13192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260620 12:22:03.148001 13192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:22:03.148084 13192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:22:03.148175 13192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:22:03.150010 13192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260620 12:22:03.150331 13192 leader_election.cc:302] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83536ce5e4274b53be4d7640b579f36a; no voters:
I20260620 12:22:03.150635 13192 leader_election.cc:288] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:22:03.150712 13196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:22:03.150957 13196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [term 2 LEADER]: Becoming Leader. State: Replica: 83536ce5e4274b53be4d7640b579f36a, State: Running, Role: LEADER
I20260620 12:22:03.151298 13196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260620 12:22:03.151808 13192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:03.152165 13196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83536ce5e4274b53be4d7640b579f36a. Latest consensus state: current_term: 2 leader_uuid: "83536ce5e4274b53be4d7640b579f36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260620 12:22:03.152356 13196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:03.152560 13197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "83536ce5e4274b53be4d7640b579f36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83536ce5e4274b53be4d7640b579f36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260620 12:22:03.152757 13197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:03.152848 13201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:03.155963 13201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:03.156442 13192 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 12:22:03.156615 13201 catalog_manager.cc:1259] Loaded cluster ID: e6244917b3c14c6eb0a5822575988cec
I20260620 12:22:03.156661 13201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:22:03.159016 13201 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:22:03.159642 13201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Loaded TSK: 0
I20260620 12:22:03.160789 13201 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:22:03.249789 13194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:03.250056 13194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:03.250164 13194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:03.254880 13194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:03.254961 13194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:03.255057 13194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:22:03.260829 13194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36107
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:03.261904 13194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:03.262931 13194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:03.273589 13230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:03.273839 13232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:03.274116 13194 server_base.cc:1034] running on GCE node
W20260620 12:22:03.273950 13229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:03.274776 13194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:03.275619 13194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:03.276809 13194 hybrid_clock.cc:648] HybridClock initialized: now 1781958123276778 us; error 47 us; skew 500 ppm
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:03.280083 13194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:22:03.281121 13194 webserver.cc:460] Webserver started at http://127.0.98.1:36107/ using document root <none> and password file <none>
I20260620 12:22:03.281669 13194 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:03.281778 13194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:03.305575 13194 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260620 12:22:03.308661 13240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.310168 13194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:22:03.310292 13194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "cdcbdaf09cdf08df685864496b3a614c"
server_key_iv: "1ce879b6d3bb0a5aa0b95b1b7cb3c9d0"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:03.311009 13194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:03.324129 13194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:03.327304 13194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:03.327517 13194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:03.328203 13194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:03.329216 13194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:03.329295 13194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.329349 13194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:03.329372 13194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.339967 13194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36785
I20260620 12:22:03.341140 13194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:22:03.347466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 13194
I20260620 12:22:03.347951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36869
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41653
--webserver_port=45449
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36937/kms with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:03.350127 13353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36785 every 8 connection(s)
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:03.366207 13357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:03.343609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.1:43679)
0620 12:22:03.344386 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:03.344437 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:03.345432 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:03.347061 (+ 1629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:03.347070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:03.351586 (+ 4516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:03.352480 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:03.352502 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.354632 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.354635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:03.356017 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:03.356028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.357369 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.357373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:03.357559 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:03.358485 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:03.358506 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:03.360082 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:03.362728 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:03.362734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:03.362748 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:03.363078 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:03.363509 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:03.363511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:03.363514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:03.363680 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:03.364435 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:03.364442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:03.364738 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 12:22:03.364920 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 12:22:03.365246 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":154,"threads_started":1}
I20260620 12:22:03.366699 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:03.343451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43679 (local address 127.0.98.62:41653)
0620 12:22:03.344096 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:03.344118 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:03.344548 (+ 430us) server_negotiation.cc:407] Connection header received
0620 12:22:03.345947 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:03.345963 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:03.346602 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:03.346938 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:03.352678 (+ 5740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.354462 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.356161 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.356917 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.357098 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:03.360324 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:03.360330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:03.360347 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:03.360380 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:03.362585 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:03.363223 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:03.363229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:03.363235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:03.363399 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:03.363806 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:03.363812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:03.363814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:03.364299 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:03.364433 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:03.365141 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 12:22:03.365444 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":116,"threads_started":1}
I20260620 12:22:03.369765 13354 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260620 12:22:03.370004 13354 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.370591 13354 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260620 12:22:03.372927 13354 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:22:03.373075 13354 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.373286 13354 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:03.374286 13354 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.374446 13354 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260620 12:22:03.375231 13354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:22:03.493428 13358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:03.493808 13358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:03.493896 13358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:03.498239 13358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:03.498319 13358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:03.498430 13358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:22:03.503991 13358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45449
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:03.505216 13358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:03.506219 13358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:03.516186 13364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:03.516261 13363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:03.516193 13366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:03.516947 13358 server_base.cc:1034] running on GCE node
I20260620 12:22:03.517383 13358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:03.518043 13358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:03.521354 13358 hybrid_clock.cc:648] HybridClock initialized: now 1781958123521318 us; error 45 us; skew 500 ppm
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:03.524278 13358 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:22:03.525314 13358 webserver.cc:460] Webserver started at http://127.0.98.2:45449/ using document root <none> and password file <none>
I20260620 12:22:03.525929 13358 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:03.526026 13358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:03.544006 13358 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:22:03.549410 13374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.551879 13358 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 12:22:03.552016 13358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "02ce9577c3c24e14bf42ac691edee9ec"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "bd3e9c31c4e5075c79a72d51d13b2598"
server_key_iv: "a727e29e9e77234ae6023f94dc16a100"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:03.552627 13358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:03.573316 13358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:03.576293 13358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:03.576493 13358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:03.577118 13358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:03.578032 13358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:03.578101 13358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.578171 13358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:03.578223 13358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.589249 13358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36869
I20260620 12:22:03.589478 13487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36869 every 8 connection(s)
I20260620 12:22:03.590654 13358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:22:03.592361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 13358
I20260620 12:22:03.592816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44153
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41653
--webserver_port=34133
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36937/kms with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:03.608426 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:03.594569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53737 (local address 127.0.98.62:41653)
0620 12:22:03.594711 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:03.594715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:03.595059 (+ 344us) server_negotiation.cc:407] Connection header received
0620 12:22:03.596338 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:03.596342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:03.596401 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:03.596482 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:03.598076 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.598860 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.600553 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.600911 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.600990 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:03.604130 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:03.604133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:03.604136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:03.604165 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:03.606012 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:03.606627 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:03.606630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:03.606632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:03.606713 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:03.607128 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:03.607132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:03.607134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:03.607363 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:03.607451 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:03.608128 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 12:22:03.608257 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:22:03.610253 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:03.593425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.2:53737)
0620 12:22:03.594871 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:03.594930 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:03.596035 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:03.596810 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:03.596819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:03.597215 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:03.597893 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:03.597911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.599022 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.599025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:03.600410 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:03.600420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.601222 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.601226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:03.601416 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:03.602225 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:03.602247 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:03.603926 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:03.606149 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:03.606155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:03.606169 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:03.606510 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:03.606810 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:03.606813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:03.606815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:03.607001 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:03.607520 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:03.607527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:03.607830 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 12:22:03.608903 (+ 1073us) client_negotiation.cc:240] Negotiation successful
0620 12:22:03.609258 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"thread_start_us":144,"threads_started":1}
I20260620 12:22:03.611744 13488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260620 12:22:03.611980 13488 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.612505 13488 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260620 12:22:03.614228 13488 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:22:03.614353 13488 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.614513 13488 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:03.615376 13488 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.615535 13488 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260620 12:22:03.616312 13488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:22:03.754674 13491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:03.754999 13491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:03.755059 13491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:03.761180 13491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:03.761265 13491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:03.761353 13491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:22:03.769336 13491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34133
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:03.770809 13491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:03.772189 13491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:03.784356 13500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:03.784358 13498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:03.784722 13491 server_base.cc:1034] running on GCE node
W20260620 12:22:03.785262 13497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:03.785668 13491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:03.786348 13491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:03.787537 13491 hybrid_clock.cc:648] HybridClock initialized: now 1781958123787516 us; error 31 us; skew 500 ppm
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:03.791252 13491 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:22:03.792364 13491 webserver.cc:460] Webserver started at http://127.0.98.3:34133/ using document root <none> and password file <none>
I20260620 12:22:03.793083 13491 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:03.793185 13491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:03.817786 13491 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:22:03.826474 13508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.828459 13491 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 12:22:03.828590 13491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "0a260735b17f4c89a2f348c6bd1a7ba1"
format_stamp: "Formatted at 2026-06-20 12:22:02 on dist-test-slave-9b3m"
server_key: "d7142e77b3e99114ee3460508e91c468"
server_key_iv: "ec0eefe154e92cac796a95acdb4fd4e4"
server_key_version: "kuduclusterkey@0"
I20260620 12:22:03.829360 13491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:03.848762 13491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:03.854442 13491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:03.854673 13491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:03.855468 13491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:03.856541 13491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:03.856608 13491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.856693 13491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:03.856726 13491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:03.871706 13491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44153
I20260620 12:22:03.873023 13491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:22:03.873922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 13491
I20260620 12:22:03.873502 13621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44153 every 8 connection(s)
Jun 20 12:22:03 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:03.892024 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:03.874664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.0.98.62:41653)
0620 12:22:03.874802 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:03.874807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:03.878715 (+ 3908us) server_negotiation.cc:407] Connection header received
0620 12:22:03.878777 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:03.878781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:03.878842 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:03.878916 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:03.881145 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.882116 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.885087 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.885499 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.885621 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:03.887735 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:03.887739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:03.887741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:03.887766 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:03.889444 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:03.889911 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:03.889915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:03.889917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:03.889998 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:03.891136 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:03.891140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:03.891141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:03.891372 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:03.891562 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:03.891743 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 12:22:03.891861 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:22:03.897554 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:03.877845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51203 (local address 127.0.98.62:41653)
0620 12:22:03.881249 (+ 3404us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:03.881259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:03.881272 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:22:03.881310 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:03.881313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:03.881356 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:03.881440 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:03.883025 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.883835 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.885424 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.885763 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.885869 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:03.893448 (+ 7579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:03.893456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:03.893459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:03.893485 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:03.895225 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:03.895735 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:03.895739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:03.895741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:03.895829 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:03.896158 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:03.896162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:03.896163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:03.896365 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:03.896468 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:03.897253 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 12:22:03.897373 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3323,"thread_start_us":97,"threads_started":1}
I20260620 12:22:03.898231 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:03.876735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.3:51203)
0620 12:22:03.878714 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:03.878771 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:03.879805 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:03.881788 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:03.881798 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:03.882146 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:03.882856 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:03.882877 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.883948 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.883952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:03.885319 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:03.885329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:03.886413 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:03.886418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:03.886629 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:03.888150 (+ 1521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:03.888171 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:03.892649 (+ 4478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:03.895339 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:03.895344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:03.895357 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:03.895604 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:03.895925 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:03.895927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:03.895929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:03.896056 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:03.896480 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:03.896487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:03.896840 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 12:22:03.896998 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 12:22:03.897331 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"thread_start_us":129,"threads_started":1}
I20260620 12:22:03.899484 13622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260620 12:22:03.899767 13622 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.900343 13622 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260620 12:22:03.902101 13622 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 12:22:03.902249 13622 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.902454 13622 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:03.903399 13622 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:03.903630 13622 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260620 12:22:03.904641 13622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 12:22:04 dist-test-slave-9b3m krb5kdc[12160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781958124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:04.375452 13354 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:04.375757 13354 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:04.376812 13156 ts_manager.cc:194] Registered new tserver with Master: b0e6d638cd1e48f4bb63ec8d9001e6d8 (127.0.98.1:36785)
I20260620 12:22:04.378383 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43679
I20260620 12:22:04.616531 13488 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:04.616858 13488 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:04.617731 13156 ts_manager.cc:194] Registered new tserver with Master: 02ce9577c3c24e14bf42ac691edee9ec (127.0.98.2:36869)
I20260620 12:22:04.618539 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53737
Jun 20 12:22:04 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:04.904891 13622 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:04.905225 13622 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260620 12:22:04.906024 13156 ts_manager.cc:194] Registered new tserver with Master: 0a260735b17f4c89a2f348c6bd1a7ba1 (127.0.98.3:44153)
I20260620 12:22:04.906870 13156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51203
I20260620 12:22:04.915225 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:22:04.933735 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:04.916800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.0.98.62:41653)
0620 12:22:04.917529 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:04.917534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:04.919821 (+ 2287us) server_negotiation.cc:407] Connection header received
0620 12:22:04.919944 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:04.919947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:04.920006 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:04.920097 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:04.921104 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.921970 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.923212 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.923547 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.923658 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:04.929464 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:04.929468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:04.929471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:04.929505 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:04.931302 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:04.931809 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:04.931813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:04.931815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:04.931905 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:04.932227 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:04.932233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:04.932235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:04.932560 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:04.932687 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:04.933416 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 12:22:04.933543 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":103,"threads_started":1}
Jun 20 12:22:04 dist-test-slave-9b3m krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 12:22:04.937934 13156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:22:04.949348 13156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:22:04.970916 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:04.961620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36785 (local address 127.0.0.1:56890)
0620 12:22:04.963503 (+ 1883us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:04.963515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:04.963605 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:04.964684 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:04.964687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:04.964707 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:04.965038 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:04.965047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.967819 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.967822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:04.969063 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:04.969075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.970376 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.970379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:04.970495 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:04.970531 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 12:22:04.970643 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 12:22:04.970756 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1826,"thread_start_us":83,"threads_started":1}
I20260620 12:22:04.971377 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:04.961275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.0.98.1:36785)
0620 12:22:04.961594 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:04.961601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:04.963627 (+ 2026us) server_negotiation.cc:407] Connection header received
0620 12:22:04.963971 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:04.963978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:04.964168 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:04.964315 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:04.965225 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.967679 (+ 2454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.969208 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.970051 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.970164 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:04.970311 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:04.971128 (+ 817us) server_negotiation.cc:299] Negotiation successful
0620 12:22:04.971239 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":149,"threads_started":1}
I20260620 12:22:04.972290 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:04.960956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36869 (local address 127.0.0.1:58832)
0620 12:22:04.962587 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:04.962635 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:04.962747 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:04.963728 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:04.963734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:04.963752 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:04.963887 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:04.963892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.966242 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.966246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:04.970440 (+ 4194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:04.970450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.971936 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.971940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:04.972028 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:04.972050 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:22:04.972093 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 12:22:04.972166 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"thread_start_us":113,"threads_started":1}
I20260620 12:22:04.973821 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:04.961120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.0.98.2:36869)
0620 12:22:04.963002 (+ 1882us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:04.963008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:04.963030 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:22:04.963099 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:04.963104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:04.963307 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:04.964656 (+ 1349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:04.964803 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.965910 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.970584 (+ 4674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.971808 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.972679 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:04.972816 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:04.972864 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 12:22:04.973683 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1783,"thread_start_us":74,"threads_started":1}
I20260620 12:22:04.974092 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:04.962303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44153 (local address 127.0.0.1:49530)
0620 12:22:04.969181 (+ 6878us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:04.969195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:04.969295 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:04.969991 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:04.969994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:04.970009 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:04.970084 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:04.970089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.971395 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.971398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:04.972486 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:04.972498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.973702 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.973706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:04.973790 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:04.973810 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:22:04.973885 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:22:04.973960 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6816,"thread_start_us":87,"threads_started":1}
I20260620 12:22:04.974639 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 682d173fc59f408bbe06495eca8992a7 (DEFAULT_TABLE table=test-table [id=8756de2627e749f3abaca31a0e308ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:04.975775 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682d173fc59f408bbe06495eca8992a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:04.976742 13422 tablet_service.cc:1467] Processing CreateTablet for tablet 682d173fc59f408bbe06495eca8992a7 (DEFAULT_TABLE table=test-table [id=8756de2627e749f3abaca31a0e308ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:04.977362 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:04.962424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.0.98.3:44153)
0620 12:22:04.966164 (+ 3740us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:04.966171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:04.969397 (+ 3226us) server_negotiation.cc:407] Connection header received
0620 12:22:04.969475 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:04.969480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:04.969694 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:04.969837 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:04.970202 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.971267 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.972639 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:04.973405 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:04.973500 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:04.973641 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:04.977130 (+ 3489us) server_negotiation.cc:299] Negotiation successful
0620 12:22:04.977229 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3631,"thread_start_us":70,"threads_started":1}
I20260620 12:22:04.977777 13422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682d173fc59f408bbe06495eca8992a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:04.979868 13540 tablet_service.cc:1467] Processing CreateTablet for tablet 682d173fc59f408bbe06495eca8992a7 (DEFAULT_TABLE table=test-table [id=8756de2627e749f3abaca31a0e308ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:04.980762 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682d173fc59f408bbe06495eca8992a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:04.998627 13656 tablet_bootstrap.cc:492] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8: Bootstrap starting.
I20260620 12:22:04.998715 13659 tablet_bootstrap.cc:492] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Bootstrap starting.
I20260620 12:22:05.001168 13656 tablet_bootstrap.cc:654] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:05.001410 13659 tablet_bootstrap.cc:654] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:05.002053 13656 log.cc:824] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:05.002249 13659 log.cc:824] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:05.002898 13658 tablet_bootstrap.cc:492] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Bootstrap starting.
I20260620 12:22:05.005682 13658 tablet_bootstrap.cc:654] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:05.006652 13658 log.cc:824] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:05.008961 13659 tablet_bootstrap.cc:492] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: No bootstrap required, opened a new log
I20260620 12:22:05.009227 13659 ts_tablet_manager.cc:1397] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260620 12:22:05.014261 13659 raft_consensus.cc:348] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.014819 13659 raft_consensus.cc:374] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:05.014921 13659 raft_consensus.cc:729] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a260735b17f4c89a2f348c6bd1a7ba1, State: Initialized, Role: FOLLOWER
I20260620 12:22:05.015529 13659 consensus_queue.cc:260] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.016713 13659 ts_tablet_manager.cc:1428] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260620 12:22:05.017127 13622 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
I20260620 12:22:05.022292 13656 tablet_bootstrap.cc:492] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8: No bootstrap required, opened a new log
I20260620 12:22:05.023669 13658 tablet_bootstrap.cc:492] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: No bootstrap required, opened a new log
I20260620 12:22:05.023944 13658 ts_tablet_manager.cc:1397] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260620 12:22:05.027438 13656 ts_tablet_manager.cc:1397] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20260620 12:22:05.031013 13658 raft_consensus.cc:348] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.036070 13658 raft_consensus.cc:374] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:05.036238 13658 raft_consensus.cc:729] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ce9577c3c24e14bf42ac691edee9ec, State: Initialized, Role: FOLLOWER
I20260620 12:22:05.036826 13658 consensus_queue.cc:260] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.037969 13658 ts_tablet_manager.cc:1428] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Time spent starting tablet: real 0.014s user 0.003s sys 0.003s
I20260620 12:22:05.038460 13656 raft_consensus.cc:348] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.038784 13656 raft_consensus.cc:374] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:05.038863 13656 raft_consensus.cc:729] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e6d638cd1e48f4bb63ec8d9001e6d8, State: Initialized, Role: FOLLOWER
I20260620 12:22:05.039611 13656 consensus_queue.cc:260] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.039921 13488 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
I20260620 12:22:05.043968 13656 ts_tablet_manager.cc:1428] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260620 12:22:05.044353 13354 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
W20260620 12:22:05.095721 13355 tablet.cc:2369] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:22:05.097744 13489 tablet.cc:2369] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:22:05.125634 13623 tablet.cc:2369] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:22:05.158613 13663 raft_consensus.cc:482] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:05.158784 13664 raft_consensus.cc:482] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:05.158893 13663 raft_consensus.cc:504] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.158978 13664 raft_consensus.cc:504] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.160080 13663 leader_election.cc:288] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ce9577c3c24e14bf42ac691edee9ec (127.0.98.2:36869), b0e6d638cd1e48f4bb63ec8d9001e6d8 (127.0.98.1:36785)
I20260620 12:22:05.160123 13664 leader_election.cc:288] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0e6d638cd1e48f4bb63ec8d9001e6d8 (127.0.98.1:36785), 0a260735b17f4c89a2f348c6bd1a7ba1 (127.0.98.3:44153)
I20260620 12:22:05.166779 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.161150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40933 (local address 127.0.98.2:36869)
0620 12:22:05.161254 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:05.161258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:05.162380 (+ 1122us) server_negotiation.cc:407] Connection header received
0620 12:22:05.162569 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:05.162572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:05.162616 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:05.162662 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:05.163064 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.163890 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.165507 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.166116 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.166193 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.166253 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:05.166584 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 12:22:05.166655 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:22:05.167344 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.160292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36869 (local address 127.0.98.3:40933)
0620 12:22:05.162296 (+ 2004us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:05.162310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:05.162455 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:05.162793 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:05.162796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:05.162812 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:05.162949 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:05.162954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.164032 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.164036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:05.165370 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:05.165380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.166344 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.166347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:05.166453 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.166475 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:22:05.167128 (+ 653us) client_negotiation.cc:240] Negotiation successful
0620 12:22:05.167206 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1943,"thread_start_us":110,"threads_started":1}
I20260620 12:22:05.168380 13442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d173fc59f408bbe06495eca8992a7" candidate_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ce9577c3c24e14bf42ac691edee9ec" is_pre_election: true
I20260620 12:22:05.168642 13442 raft_consensus.cc:2454] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a260735b17f4c89a2f348c6bd1a7ba1 in term 0.
I20260620 12:22:05.169157 13512 leader_election.cc:302] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ce9577c3c24e14bf42ac691edee9ec, 0a260735b17f4c89a2f348c6bd1a7ba1; no voters:
I20260620 12:22:05.169436 13663 raft_consensus.cc:2781] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:22:05.169525 13663 raft_consensus.cc:482] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:22:05.169562 13663 raft_consensus.cc:3037] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:05.170898 13663 raft_consensus.cc:504] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.171263 13663 leader_election.cc:288] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [CANDIDATE]: Term 1 election: Requested vote from peers 02ce9577c3c24e14bf42ac691edee9ec (127.0.98.2:36869), b0e6d638cd1e48f4bb63ec8d9001e6d8 (127.0.98.1:36785)
I20260620 12:22:05.171734 13442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d173fc59f408bbe06495eca8992a7" candidate_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ce9577c3c24e14bf42ac691edee9ec"
I20260620 12:22:05.171850 13442 raft_consensus.cc:3037] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:05.172750 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.160763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52169 (local address 127.0.98.1:36785)
0620 12:22:05.160917 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:05.160924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:05.168278 (+ 7354us) server_negotiation.cc:407] Connection header received
0620 12:22:05.168448 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:05.168451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:05.168501 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:05.168547 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:05.169126 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.169824 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.171437 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.172012 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.172116 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.172184 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:05.172551 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 12:22:05.172634 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 12:22:05.173218 13442 raft_consensus.cc:2454] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a260735b17f4c89a2f348c6bd1a7ba1 in term 1.
I20260620 12:22:05.173517 13512 leader_election.cc:302] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ce9577c3c24e14bf42ac691edee9ec, 0a260735b17f4c89a2f348c6bd1a7ba1; no voters:
I20260620 12:22:05.173703 13663 raft_consensus.cc:2781] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:05.173918 13663 raft_consensus.cc:686] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 0a260735b17f4c89a2f348c6bd1a7ba1, State: Running, Role: LEADER
I20260620 12:22:05.174297 13663 consensus_queue.cc:237] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260620 12:22:05.177433 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.160348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36785 (local address 127.0.98.3:52169)
0620 12:22:05.168206 (+ 7858us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:05.168220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:05.168341 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:05.168804 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:05.168807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:05.168825 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:05.168894 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:05.168899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.170068 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.170070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:05.171252 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:05.171262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.172329 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.172332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:05.172416 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.172441 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:22:05.173080 (+ 639us) client_negotiation.cc:240] Negotiation successful
0620 12:22:05.173184 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7805,"thread_start_us":95,"threads_started":1}
I20260620 12:22:05.177878 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.160860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58219 (local address 127.0.98.1:36785)
0620 12:22:05.162163 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:05.162171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:05.173354 (+ 11183us) server_negotiation.cc:407] Connection header received
0620 12:22:05.173481 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:05.173484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:05.173534 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:05.173686 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:05.174220 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.174979 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.176396 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.176963 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.177036 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.177125 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:05.177677 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 12:22:05.177749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":130,"threads_started":1}
I20260620 12:22:05.177564 13156 catalog_manager.cc:5495] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a260735b17f4c89a2f348c6bd1a7ba1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } health_report { overall_health: HEALTHY } } }
I20260620 12:22:05.178882 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d173fc59f408bbe06495eca8992a7" candidate_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8"
I20260620 12:22:05.179100 13308 raft_consensus.cc:3037] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:05.180424 13308 raft_consensus.cc:2454] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a260735b17f4c89a2f348c6bd1a7ba1 in term 1.
I20260620 12:22:05.178900 13301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d173fc59f408bbe06495eca8992a7" candidate_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" is_pre_election: true
I20260620 12:22:05.180835 13301 raft_consensus.cc:2362] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a260735b17f4c89a2f348c6bd1a7ba1 in term 1. Re-sending same reply.
I20260620 12:22:05.181990 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.160443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36785 (local address 127.0.98.2:58219)
0620 12:22:05.173266 (+ 12823us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:05.173282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:05.173390 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:05.173887 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:05.173893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:05.173915 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:05.174086 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:05.174093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.175131 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.175134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:05.176259 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:05.176269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.177219 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.177223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:05.177315 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.177336 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:22:05.181525 (+ 4189us) client_negotiation.cc:240] Negotiation successful
0620 12:22:05.181822 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12754,"thread_start_us":108,"threads_started":1}
I20260620 12:22:05.182909 13301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d173fc59f408bbe06495eca8992a7" candidate_uuid: "02ce9577c3c24e14bf42ac691edee9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" is_pre_election: true
I20260620 12:22:05.183091 13301 raft_consensus.cc:2379] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02ce9577c3c24e14bf42ac691edee9ec in current term 1: Already voted for candidate 0a260735b17f4c89a2f348c6bd1a7ba1 in this term.
I20260620 12:22:05.188292 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.161873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50937 (local address 127.0.98.3:44153)
0620 12:22:05.162000 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:05.162006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:05.181421 (+ 19415us) server_negotiation.cc:407] Connection header received
0620 12:22:05.181606 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:05.181610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:05.181661 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:05.181714 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:05.182071 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.182780 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.184395 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.184944 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.185071 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.185138 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:05.188106 (+ 2968us) server_negotiation.cc:299] Negotiation successful
0620 12:22:05.188183 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:22:05.188671 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.161565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44153 (local address 127.0.98.2:50937)
0620 12:22:05.181335 (+ 19770us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:05.181351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:05.181485 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:05.181830 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:05.181832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:05.181849 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:05.181945 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:05.181951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.183440 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.183444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:05.184264 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:05.184273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.187815 (+ 3542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.187819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:05.187931 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.187956 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:22:05.188464 (+ 508us) client_negotiation.cc:240] Negotiation successful
0620 12:22:05.188537 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19703,"thread_start_us":83,"threads_started":1}
I20260620 12:22:05.189193 13576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682d173fc59f408bbe06495eca8992a7" candidate_uuid: "02ce9577c3c24e14bf42ac691edee9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a260735b17f4c89a2f348c6bd1a7ba1" is_pre_election: true
I20260620 12:22:05.189853 13378 leader_election.cc:302] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02ce9577c3c24e14bf42ac691edee9ec; no voters: 0a260735b17f4c89a2f348c6bd1a7ba1, b0e6d638cd1e48f4bb63ec8d9001e6d8
I20260620 12:22:05.190297 13664 raft_consensus.cc:2726] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 12:22:05.210105 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.205210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.0.98.3:44153)
0620 12:22:05.205326 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:05.205331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:05.205400 (+ 69us) server_negotiation.cc:407] Connection header received
0620 12:22:05.205582 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:05.205586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:05.205674 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:05.205774 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:22:05.206075 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.206817 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.207965 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:05.208316 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:05.208415 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:05.208820 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:22:05.209416 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:22:05.209529 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:05.209902 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 12:22:05.209957 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:22:05.216830 13301 raft_consensus.cc:1261] T 682d173fc59f408bbe06495eca8992a7 P b0e6d638cd1e48f4bb63ec8d9001e6d8 [term 1 FOLLOWER]: Refusing update from remote peer 0a260735b17f4c89a2f348c6bd1a7ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:05.217031 13442 raft_consensus.cc:1261] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 1 FOLLOWER]: Refusing update from remote peer 0a260735b17f4c89a2f348c6bd1a7ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:05.217530 13671 consensus_queue.cc:1035] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e6d638cd1e48f4bb63ec8d9001e6d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:05.217710 13663 consensus_queue.cc:1035] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ce9577c3c24e14bf42ac691edee9ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:05.228379 13674 mvcc.cc:205] Tried to move back new op lower bound from 7298900480880484352 to 7298900480717025280. Current Snapshot: MvccSnapshot[applied={T|T < 7298900480880484352}]
I20260620 12:22:05.234401 13676 mvcc.cc:205] Tried to move back new op lower bound from 7298900480880484352 to 7298900480717025280. Current Snapshot: MvccSnapshot[applied={T|T < 7298900480880484352}]
I20260620 12:22:05.240139 13155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:05.240309 13155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:05.242815 13155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83536ce5e4274b53be4d7640b579f36a: Sending DeleteTablet for 3 replicas of tablet 682d173fc59f408bbe06495eca8992a7
I20260620 12:22:05.243695 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet 682d173fc59f408bbe06495eca8992a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56890
I20260620 12:22:05.244554 13422 tablet_service.cc:1514] Processing DeleteTablet for tablet 682d173fc59f408bbe06495eca8992a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58832
I20260620 12:22:05.245199 13540 tablet_service.cc:1514] Processing DeleteTablet for tablet 682d173fc59f408bbe06495eca8992a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49530
I20260620 12:22:05.246274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 13194
I20260620 12:22:05.252070 13684 tablet_replica.cc:330] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: stopping tablet replica
I20260620 12:22:05.253090 13684 raft_consensus.cc:2229] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:05.253418 13685 tablet_replica.cc:330] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: stopping tablet replica
I20260620 12:22:05.254285 13684 raft_consensus.cc:2258] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:05.255425 13685 raft_consensus.cc:2229] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:22:05.255868 13685 raft_consensus.cc:2258] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:05.257625 13684 ts_tablet_manager.cc:1905] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:05.260530 13684 ts_tablet_manager.cc:1918] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:05.260668 13684 log.cc:1196] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/682d173fc59f408bbe06495eca8992a7
I20260620 12:22:05.261018 13684 ts_tablet_manager.cc:1939] T 682d173fc59f408bbe06495eca8992a7 P 02ce9577c3c24e14bf42ac691edee9ec: Deleting consensus metadata
I20260620 12:22:05.262413 13143 catalog_manager.cc:4841] TS 02ce9577c3c24e14bf42ac691edee9ec (127.0.98.2:36869): tablet 682d173fc59f408bbe06495eca8992a7 (table test-table [id=8756de2627e749f3abaca31a0e308ef5]) successfully deleted
I20260620 12:22:05.267407 13685 ts_tablet_manager.cc:1905] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:05.270170 13685 ts_tablet_manager.cc:1918] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:05.270277 13685 log.cc:1196] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/682d173fc59f408bbe06495eca8992a7
I20260620 12:22:05.270620 13685 ts_tablet_manager.cc:1939] T 682d173fc59f408bbe06495eca8992a7 P 0a260735b17f4c89a2f348c6bd1a7ba1: Deleting consensus metadata
I20260620 12:22:05.271839 13143 catalog_manager.cc:4841] TS 0a260735b17f4c89a2f348c6bd1a7ba1 (127.0.98.3:44153): tablet 682d173fc59f408bbe06495eca8992a7 (table test-table [id=8756de2627e749f3abaca31a0e308ef5]) successfully deleted
W20260620 12:22:05.272423 13140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36785 (error 108)
I20260620 12:22:05.273479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 13358
I20260620 12:22:05.282083 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:05.281711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36785 (local address 127.0.0.1:56894)
0620 12:22:05.281916 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:05.281988 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 12:22:05.282282 13140 catalog_manager.cc:4584] TS b0e6d638cd1e48f4bb63ec8d9001e6d8 (127.0.98.1:36785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 682d173fc59f408bbe06495eca8992a7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36785: connect: Connection refused (error 111)
I20260620 12:22:05.284381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 13491
I20260620 12:22:05.291520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 13123
I20260620 12:22:05.298240 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 12:22:05.643381 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 12:22:05.643489 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 12:22:05.987965 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 12:22:05.988 UTC [12190] LOG: received smart shutdown request
2026-06-20 12:22:05.992 UTC [12190] LOG: background worker "logical replication launcher" (PID 12199) exited with exit code 1
2026-06-20 12:22:05.992 UTC [12194] LOG: shutting down
2026-06-20 12:22:06.154 UTC [12190] LOG: database system is shut down
2026-06-20T12:22:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48850 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:22:06 dist-test-slave-9b3m krb5kdc[13697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:22:06 dist-test-slave-9b3m krb5kdc[13697](info): set up 2 sockets
Jun 20 12:22:06 dist-test-slave-9b3m krb5kdc[13697](info): commencing operation
krb5kdc: starting...
W20260620 12:22:07.313337 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:22:07 dist-test-slave-9b3m krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:22:07.433588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40307,127.0.98.61:44165,127.0.98.60:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:07.540026 13710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:07.540268 13710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:07.540307 13710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:07.544149 13710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:07.544193 13710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:07.544212 13710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:07.544229 13710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:07.544243 13710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:07.549271 13710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40307,127.0.98.61:44165,127.0.98.60:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:07.550216 13710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:07.551139 13710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:07.559674 13716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:07.559687 13715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:07.559732 13710 server_base.cc:1034] running on GCE node
W20260620 12:22:07.559799 13718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:07.560563 13710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:07.561739 13710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:07.562933 13710 hybrid_clock.cc:648] HybridClock initialized: now 1781958127562913 us; error 33 us; skew 500 ppm
Jun 20 12:22:07 dist-test-slave-9b3m krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:07.565747 13710 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:07.566610 13710 webserver.cc:460] Webserver started at http://127.0.98.62:44813/ using document root <none> and password file <none>
I20260620 12:22:07.567112 13710 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:07.567168 13710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:07.567322 13710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:07.569350 13710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "8ff863d658014193b5c6a1f482ac0330"
format_stamp: "Formatted at 2026-06-20 12:22:07 on dist-test-slave-9b3m"
server_key: "8165c4430bff567ee7bca09a69858a61"
server_key_iv: "59931dfd4e4c9a343f43db7a957dcfd8"
server_key_version: "encryptionkey@0"
I20260620 12:22:07.569850 13710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ff863d658014193b5c6a1f482ac0330"
format_stamp: "Formatted at 2026-06-20 12:22:07 on dist-test-slave-9b3m"
server_key: "8165c4430bff567ee7bca09a69858a61"
server_key_iv: "59931dfd4e4c9a343f43db7a957dcfd8"
server_key_version: "encryptionkey@0"
I20260620 12:22:07.573383 13710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:22:07.575651 13725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:07.576996 13710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:07.577147 13710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "8ff863d658014193b5c6a1f482ac0330"
format_stamp: "Formatted at 2026-06-20 12:22:07 on dist-test-slave-9b3m"
server_key: "8165c4430bff567ee7bca09a69858a61"
server_key_iv: "59931dfd4e4c9a343f43db7a957dcfd8"
server_key_version: "encryptionkey@0"
I20260620 12:22:07.577255 13710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:07.608660 13710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:07.611745 13710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:07.611904 13710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:07.619150 13710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40307
I20260620 12:22:07.619176 13777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40307 every 8 connection(s)
I20260620 12:22:07.620079 13710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:07.622746 13778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:07.626456 13778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:22:07.629715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 13710
I20260620 12:22:07.629858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:22:07.630151 392 external_mini_cluster.cc:1428] Setting key ab4fee6921d57c54cd968ab043afa04b
Jun 20 12:22:07 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 12:22:07.641834 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.628652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.98.62:40307)
0620 12:22:07.629111 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:07.629125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:07.629353 (+ 228us) server_negotiation.cc:407] Connection header received
0620 12:22:07.630501 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:07.630510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:07.630740 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:07.630947 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:07.631719 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.632641 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.633672 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.634302 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.634471 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:07.636785 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:07.636793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:07.636805 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:07.636834 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:07.638708 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:07.639159 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:07.639164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:07.639165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:07.639261 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:07.639559 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:07.639562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:07.639563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:07.639950 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:07.640102 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:07.640505 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 12:22:07.640863 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":296,"thread_start_us":178,"threads_started":1}
I20260620 12:22:07.641830 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.628652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40307 (local address 127.0.0.1:51860)
0620 12:22:07.629182 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.629244 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:07.630154 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:07.630971 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:07.630976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:07.631317 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:07.631562 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:07.631574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.632763 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.632766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:07.633554 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:07.633563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.634450 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.634453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:07.634543 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:07.635177 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:07.635194 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:07.636635 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:07.638816 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:07.638824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:07.638837 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:07.639050 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:07.639351 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:07.639353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:07.639354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:07.639455 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:07.640120 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:07.640126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:07.640247 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 12:22:07.640464 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 12:22:07.640912 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":166,"spinlock_wait_cycles":119808,"thread_start_us":210,"threads_started":1}
I20260620 12:22:07.644373 13778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:22:07.645109 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.644833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50102)
0620 12:22:07.644943 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.645004 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 12:22:07.645305 13727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
I20260620 12:22:07.646124 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.645925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50106)
0620 12:22:07.646027 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.646072 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 12:22:07.646253 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44165: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 12:22:07.656466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44165
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40307,127.0.98.61:44165,127.0.98.60:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:07.695505 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } attempt: 1
I20260620 12:22:07.696367 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.696062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50114)
0620 12:22:07.696223 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.696290 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 12:22:07.697145 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.696859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50124)
0620 12:22:07.696968 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.697021 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 12:22:07.697271 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44165: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
W20260620 12:22:07.764964 13785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:07.765270 13785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:07.765347 13785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 12:22:07.765458 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } attempt: 2
I20260620 12:22:07.766295 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.766005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50128)
0620 12:22:07.766145 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.766206 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 12:22:07.766981 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.766748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50130)
0620 12:22:07.766874 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.766927 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 12:22:07.767103 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44165: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44165: connect: Connection refused (error 111)
W20260620 12:22:07.769640 13785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:07.769722 13785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:07.769771 13785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:07.769812 13785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:07.769848 13785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:07.774958 13785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40307,127.0.98.61:44165,127.0.98.60:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:07.775983 13785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:07.776983 13785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:07.785233 13793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:07.785216 13790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:07.785573 13785 server_base.cc:1034] running on GCE node
W20260620 12:22:07.785211 13791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:07.786144 13785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:07.787209 13785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:07.788419 13785 hybrid_clock.cc:648] HybridClock initialized: now 1781958127788400 us; error 24 us; skew 500 ppm
Jun 20 12:22:07 dist-test-slave-9b3m krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:07.791275 13785 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 12:22:07.792146 13785 webserver.cc:460] Webserver started at http://127.0.98.61:33701/ using document root <none> and password file <none>
I20260620 12:22:07.792629 13785 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:07.792685 13785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:07.792896 13785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:07.794881 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/data/instance:
uuid: "9e768fdcc85a4afc9864ce6c1ff0b692"
format_stamp: "Formatted at 2026-06-20 12:22:07 on dist-test-slave-9b3m"
server_key: "f95c7bc2fd54c5922d060414673e5e56"
server_key_iv: "cce9a31c75654b6b7a3dd701a68a978f"
server_key_version: "encryptionkey@0"
I20260620 12:22:07.795423 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/wal/instance:
uuid: "9e768fdcc85a4afc9864ce6c1ff0b692"
format_stamp: "Formatted at 2026-06-20 12:22:07 on dist-test-slave-9b3m"
server_key: "f95c7bc2fd54c5922d060414673e5e56"
server_key_iv: "cce9a31c75654b6b7a3dd701a68a978f"
server_key_version: "encryptionkey@0"
I20260620 12:22:07.798979 13785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:22:07.801316 13800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:07.802587 13785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:22:07.802718 13785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/wal
uuid: "9e768fdcc85a4afc9864ce6c1ff0b692"
format_stamp: "Formatted at 2026-06-20 12:22:07 on dist-test-slave-9b3m"
server_key: "f95c7bc2fd54c5922d060414673e5e56"
server_key_iv: "cce9a31c75654b6b7a3dd701a68a978f"
server_key_version: "encryptionkey@0"
I20260620 12:22:07.802829 13785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:07.814554 13785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:07.817648 13785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:07.817843 13785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:07.824647 13785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44165
I20260620 12:22:07.824700 13852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44165 every 8 connection(s)
I20260620 12:22:07.825613 13785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:22:07.828296 13853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:07.832060 13853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:22:07.832293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 13785
I20260620 12:22:07.832516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-1/wal/instance
I20260620 12:22:07.832806 392 external_mini_cluster.cc:1428] Setting key d37651e8d77eefb8072c2e3e4d14747c
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 12:22:07 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:07.847384 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.834134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.0.98.62:40307)
0620 12:22:07.834301 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:07.834308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:07.835254 (+ 946us) server_negotiation.cc:407] Connection header received
0620 12:22:07.836245 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:07.836249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:07.836315 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:07.836412 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:07.837639 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.838164 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.839266 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.839501 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.839610 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:07.843506 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:07.843509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:07.843511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:07.843533 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:07.845316 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:07.845832 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:07.845835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:07.845837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:07.845920 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:07.846255 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:07.846259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:07.846261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:07.846445 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:07.846551 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:07.847054 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 12:22:07.847201 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:22:07.848111 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.834557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40307 (local address 127.0.0.1:51876)
0620 12:22:07.835058 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.835115 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:07.836026 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:07.836571 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:07.836598 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:07.836978 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:07.837495 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:07.837508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.838296 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.838304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:07.839140 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:07.839149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.839645 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.839649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:07.839813 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:07.840430 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:07.840445 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:07.843333 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:07.845440 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:07.845446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:07.845459 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:07.845718 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:07.846014 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:07.846018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:07.846020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:07.846152 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:07.846550 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:07.846556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:07.846808 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 12:22:07.846995 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 12:22:07.847305 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
I20260620 12:22:07.849817 13853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:22:07 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:22:07.858253 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } attempt: 3
I20260620 12:22:07.859347 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.850341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50136)
0620 12:22:07.850493 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.850504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:07.850562 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:07.851122 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:07.851125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:07.851290 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:07.851372 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:07.851377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.852371 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.852377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:07.852987 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:07.852995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.853652 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.853656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:07.853741 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:07.854220 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:07.854231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:07.855495 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:07.857657 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:07.857659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:07.857661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:07.857843 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:07.858179 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:07.858181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:07.858182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:07.858254 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:07.858901 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:07.858907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:07.859000 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 12:22:07.859126 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 12:22:07.859235 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 12:22:07.859592 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.850473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.0.98.61:44165)
0620 12:22:07.850809 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:07.850818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:07.850835 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:22:07.850865 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:07.850869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:07.851006 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:07.851143 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:07.851478 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.852260 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.853152 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.853537 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.853656 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:07.855619 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:07.855627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:07.855637 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:07.855659 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:07.857543 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:07.857939 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:07.857943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:07.857944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:07.858095 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:07.858371 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:07.858377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:07.858382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:07.858696 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:07.858926 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:07.859232 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 12:22:07.859350 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":258,"thread_start_us":160,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 12:22:07.861220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45941
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40307,127.0.98.61:44165,127.0.98.60:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:07.861686 13853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:22:07.862501 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.862224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40722)
0620 12:22:07.862344 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.862421 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 12:22:07.863317 13803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
I20260620 12:22:07.864245 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.864055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40738)
0620 12:22:07.864145 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.864189 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 12:22:07.864450 13853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45941: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Jun 20 12:22:07 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:22:07.870169 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.858700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50140)
0620 12:22:07.858859 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.858876 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:07.858951 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:07.859849 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:07.859852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:07.860016 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:07.860644 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:07.860655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.861695 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.861702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:07.862678 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:07.862689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.863579 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.863586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:07.863694 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:07.864160 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:07.864179 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:07.865869 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:07.868240 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:07.868245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:07.868248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:07.868521 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:07.868895 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:07.868899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:07.868903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:07.869002 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:07.869542 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:07.869547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:07.869666 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 12:22:07.869763 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 12:22:07.869935 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 12:22:07.870962 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.858739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.98.61:44165)
0620 12:22:07.859622 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:07.859631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:07.859646 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:22:07.859684 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:07.859689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:07.859735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:07.859821 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:07.860777 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.861580 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.863007 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:07.863427 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:07.863975 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:07.866060 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:07.866066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:07.866072 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:07.866101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:07.868098 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:07.868658 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:07.868663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:07.868666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:07.868758 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:07.869160 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:07.869165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:07.869168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:07.869413 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:07.870494 (+ 1081us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:07.870581 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 12:22:07.870711 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":105,"threads_started":1}
I20260620 12:22:07.872023 13778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:22:07.872814 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.872490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40748)
0620 12:22:07.872642 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.872712 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 12:22:07.873626 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.873296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40758)
0620 12:22:07.873485 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.873541 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 12:22:07.873836 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45941: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
I20260620 12:22:07.905031 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } attempt: 1
I20260620 12:22:07.906045 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.905646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40762)
0620 12:22:07.905840 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.905946 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 12:22:07.906814 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.906600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40770)
0620 12:22:07.906698 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.906753 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 12:22:07.906958 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45941: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
I20260620 12:22:07.913810 13853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } attempt: 1
I20260620 12:22:07.914544 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.914261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40784)
0620 12:22:07.914402 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.914467 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 12:22:07.915266 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.915041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40792)
0620 12:22:07.915149 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.915200 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 12:22:07.915388 13853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45941: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
W20260620 12:22:07.978554 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:07.978811 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:07.978849 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 12:22:07.982102 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } attempt: 2
I20260620 12:22:07.982885 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.982563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40794)
0620 12:22:07.982724 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.982788 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:22:07.983575 13853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } attempt: 2
I20260620 12:22:07.983676 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.983405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40798)
0620 12:22:07.983505 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.983608 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 12:22:07.983803 13860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:07.983860 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:07.983831 13778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45941: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
W20260620 12:22:07.983901 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:07.983937 13860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:07.983964 13860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:07.984249 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.983984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40808)
0620 12:22:07.984114 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.984172 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 12:22:07.984952 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:07.984726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40820)
0620 12:22:07.984843 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:07.984893 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 12:22:07.985144 13853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45941: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45941: connect: Connection refused (error 111)
I20260620 12:22:07.989748 13860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40307,127.0.98.61:44165,127.0.98.60:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:07.990762 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:07.991777 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:08.000011 13869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:08.000036 13866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:08.000188 13860 server_base.cc:1034] running on GCE node
W20260620 12:22:08.000061 13867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:08.000831 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:08.001922 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:08.003124 13860 hybrid_clock.cc:648] HybridClock initialized: now 1781958128003092 us; error 44 us; skew 500 ppm
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:08.005988 13860 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 12:22:08.007007 13860 webserver.cc:460] Webserver started at http://127.0.98.60:37695/ using document root <none> and password file <none>
I20260620 12:22:08.007550 13860 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:08.007640 13860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:08.007869 13860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:08.009919 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/data/instance:
uuid: "156ec9e5caf04f6e97ee958e76f05ca3"
format_stamp: "Formatted at 2026-06-20 12:22:08 on dist-test-slave-9b3m"
server_key: "a789db7f1605e60fbff693c364b7cac0"
server_key_iv: "cefa752053f7a687a92e926e0260451e"
server_key_version: "encryptionkey@0"
I20260620 12:22:08.010465 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/wal/instance:
uuid: "156ec9e5caf04f6e97ee958e76f05ca3"
format_stamp: "Formatted at 2026-06-20 12:22:08 on dist-test-slave-9b3m"
server_key: "a789db7f1605e60fbff693c364b7cac0"
server_key_iv: "cefa752053f7a687a92e926e0260451e"
server_key_version: "encryptionkey@0"
I20260620 12:22:08.014047 13860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:22:08.016345 13876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:08.017717 13860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:08.017901 13860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/wal
uuid: "156ec9e5caf04f6e97ee958e76f05ca3"
format_stamp: "Formatted at 2026-06-20 12:22:08 on dist-test-slave-9b3m"
server_key: "a789db7f1605e60fbff693c364b7cac0"
server_key_iv: "cefa752053f7a687a92e926e0260451e"
server_key_version: "encryptionkey@0"
I20260620 12:22:08.018013 13860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:08.040102 13860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:08.043979 13860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:08.044183 13860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:08.052129 13860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45941
I20260620 12:22:08.052147 13928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45941 every 8 connection(s)
I20260620 12:22:08.053135 13860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:22:08.055863 13929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:08.058501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 13860
I20260620 12:22:08.058657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781957967886955-392-0/minicluster-data/master-2/wal/instance
I20260620 12:22:08.058939 392 external_mini_cluster.cc:1428] Setting key 8da3f1553c2fcc2595dcb9e94e9de0ea
I20260620 12:22:08.059628 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:22:08.071976 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.061492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.0.98.62:40307)
0620 12:22:08.061634 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.061638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.061924 (+ 286us) server_negotiation.cc:407] Connection header received
0620 12:22:08.062143 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.062146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.062207 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.062304 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.063195 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.063709 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.064778 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.064989 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.065132 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.067739 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.067745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.067749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.067781 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.069511 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.070152 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.070154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.070156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.070252 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.070571 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.070577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.070581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.070843 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.070947 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.071614 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.071741 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:22:08.072878 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.061420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40307 (local address 127.0.0.1:51882)
0620 12:22:08.061780 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:08.061829 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:08.061951 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:08.062360 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:08.062365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:08.062733 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:08.063043 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.063055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.063819 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.063823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.064613 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.064622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.065185 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.065192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.065373 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.066066 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:08.066083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:08.067520 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:08.069623 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.069630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.069639 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.070060 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.070343 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.070345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.070347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.070470 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.070946 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:08.070951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:08.071366 (+ 415us) client_negotiation.cc:769] Sending connection context
0620 12:22:08.071532 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 12:22:08.071895 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":53,"spinlock_wait_cycles":3328,"thread_start_us":121,"threads_started":1}
I20260620 12:22:08.072878 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.060280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.0.98.60:45941)
0620 12:22:08.060702 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.060718 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.060752 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:22:08.061508 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.061531 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.061951 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.062204 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.062941 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.063904 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.064728 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.065166 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.065359 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.068138 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.068146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.068160 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.068197 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.070162 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.070629 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.070633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.070634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.070707 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.071097 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.071100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.071101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.071366 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.071539 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.071784 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.071939 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":1280,"thread_start_us":120,"threads_started":1}
I20260620 12:22:08.074092 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:22:08.076318 13853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } attempt: 3
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:22:08.082988 13778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } attempt: 3
I20260620 12:22:08.084416 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.074811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.0.98.61:44165)
0620 12:22:08.074945 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.074951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.074968 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:22:08.075003 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.075004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.075038 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.075135 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.075683 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.076330 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.077398 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.077785 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.077857 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.080467 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.080470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.080473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.080499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.082353 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.082875 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.082879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.082882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.082967 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.083285 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.083291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.083296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.083577 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.083666 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.084084 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.084218 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:22:08.084905 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.074613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44165 (local address 127.0.0.1:50144)
0620 12:22:08.074758 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:08.074774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:08.074857 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:08.075154 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:08.075157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:08.075431 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:08.075557 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.075566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.076442 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.076446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.077267 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.077278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.078001 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.078004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.078114 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.078830 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:08.078849 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:08.080252 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:08.082480 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.082483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.082484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.082754 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.083085 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.083088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.083090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.083176 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.083795 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:08.083798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:08.083886 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 12:22:08.084599 (+ 713us) client_negotiation.cc:240] Negotiation successful
0620 12:22:08.084730 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 12:22:08.085604 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:22:08.086459 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.076690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40836)
0620 12:22:08.076804 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:08.076818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:08.076888 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:08.077356 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:08.077359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:08.077514 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:08.077597 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.077601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.078414 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.078417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.078971 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.078979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.079401 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.079405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.079489 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.079957 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:08.079974 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:08.082158 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:08.084465 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.084469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.084471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.084722 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.085449 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.085453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.085455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.085545 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.085999 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:08.086002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:08.086083 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 12:22:08.086170 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:22:08.086310 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 12:22:08.086978 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.075167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.98.62:40307)
0620 12:22:08.075270 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.075273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.075503 (+ 230us) server_negotiation.cc:407] Connection header received
0620 12:22:08.075599 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.075601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.075662 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.075743 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.076263 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.077035 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.078085 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.078426 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.078516 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.082148 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.082153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.082157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.082184 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.084646 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.085169 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.085173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.085175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.085256 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.085568 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.085571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.085573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.085811 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.086700 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.086793 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.086879 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 12:22:08.087419 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.076736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.0.98.60:45941)
0620 12:22:08.077176 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.077180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.077191 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:22:08.077226 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.077228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.077257 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.077339 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.077716 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.078290 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.079085 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.079297 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.079398 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.082284 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.082287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.082289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.082311 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.084379 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.084832 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.084837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.084839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.084923 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.085656 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.085659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.085661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.085895 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.086882 (+ 987us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.086953 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.087251 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:22:08.091362 13853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692: Bootstrap starting.
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:22:08.093557 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.083349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40840)
0620 12:22:08.083469 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:08.083484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:08.083554 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:08.084169 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:08.084172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:08.084396 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:08.084504 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.084514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.086188 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.086191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.086982 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.086991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.087607 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.087611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.087700 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.088382 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:08.088399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:08.089926 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:08.091984 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.091987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.091989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.092212 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.092522 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.092525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.092527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.092610 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.093090 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:08.093093 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:08.093183 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 12:22:08.093265 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 12:22:08.093398 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 12:22:08.094172 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.083552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.0.98.60:45941)
0620 12:22:08.083929 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.083940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.083954 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:22:08.083986 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.083989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.084028 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.084293 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.084860 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.086064 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.087125 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.087477 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.090107 (+ 2630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.090247 (+ 140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.090250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.090253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.090269 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.091861 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.092331 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.092334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.092336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.092422 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.092732 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.092735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.092737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.092946 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.093835 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.093906 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.094012 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":305,"thread_start_us":134,"threads_started":1}
I20260620 12:22:08.094966 13853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:08.095649 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.086002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45941 (local address 127.0.0.1:40856)
0620 12:22:08.086311 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:08.086326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:08.086398 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:08.087790 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:08.087792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:08.087984 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:08.088056 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.088063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.088907 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.088910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.089555 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:08.089564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.090029 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.090032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:08.090132 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.090586 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:08.090602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:08.091968 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:08.093947 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.093953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.093956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.094206 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.094695 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:08.094701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:08.094704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:08.094794 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:08.095231 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:08.095233 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:08.095301 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 12:22:08.095380 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:22:08.095496 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":25,"thread_start_us":99,"threads_started":1}
I20260620 12:22:08.095839 13853 log.cc:824] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:08.096259 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.086585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.98.60:45941)
0620 12:22:08.087247 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.087255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.087267 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:22:08.087542 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.087548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.087595 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.087675 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.088173 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.088792 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.089695 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.089928 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.090031 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.092096 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.092099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.092102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.092122 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.093586 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.094521 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.094523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.094525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.094606 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.094882 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.094885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.094886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.095118 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.095201 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.096005 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.096104 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":604,"thread_start_us":94,"threads_started":1}
I20260620 12:22:08.096967 13778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330: Bootstrap starting.
I20260620 12:22:08.098680 13853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692: No bootstrap required, opened a new log
I20260620 12:22:08.099215 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3: Bootstrap starting.
I20260620 12:22:08.099537 13778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:08.100405 13778 log.cc:824] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:22:08 dist-test-slave-9b3m krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:22:08.102774 13853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.102986 13853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:08.103041 13853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e768fdcc85a4afc9864ce6c1ff0b692, State: Initialized, Role: FOLLOWER
I20260620 12:22:08.103235 13778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330: No bootstrap required, opened a new log
I20260620 12:22:08.103519 13853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.104800 13853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:08.105115 13929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:08.106038 13929 log.cc:824] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:08.107569 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.093449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.0.98.61:44165)
0620 12:22:08.093594 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.093597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.093613 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:22:08.093676 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.093680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.093731 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.093803 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.094655 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.095431 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.096498 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.096818 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.096912 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.103416 (+ 6504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.103419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.103422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.103447 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.105012 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.105581 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.105585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.105586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.105667 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.105967 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.105971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.105973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.106209 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.107159 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.107235 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.107332 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 12:22:08.108474 13778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.108924 13778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:08.108985 13778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff863d658014193b5c6a1f482ac0330, State: Initialized, Role: FOLLOWER
I20260620 12:22:08.109354 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3: No bootstrap required, opened a new log
I20260620 12:22:08.110152 13778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.111347 13778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:08.109148 13785 master_runner.cc:186] Error getting master registration for 127.0.98.62:40307: OK, instance_id { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" instance_seqno: 1781958127617060 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:22:08.112361 13785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 12:22:08.112442 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:22:08.112558 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 12:22:08.113634 13731 tablet.cc:2369] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:22:08.113744 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.113867 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:22:08.114049 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.114173 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:22:08.114943 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 12:22:08.116986 13964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:22:08.117076 13964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:22:08.117126 13929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.117365 13929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:08.117424 13929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156ec9e5caf04f6e97ee958e76f05ca3, State: Initialized, Role: FOLLOWER
I20260620 12:22:08.117895 13929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.119189 13929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:08.121624 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.121745 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:22:08.118321 13710 master_runner.cc:186] Error getting master registration for 127.0.98.60:45941: OK, instance_id { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" instance_seqno: 1781958128050108 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:22:08.122560 13710 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:22:08.126297 13860 master_runner.cc:263] No leader master found from master 156ec9e5caf04f6e97ee958e76f05ca3
I20260620 12:22:08.126362 13860 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 12:22:08.127956 13981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:22:08.128050 13981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:22:08.138803 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.125899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.0.98.61:44165)
0620 12:22:08.125997 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.126001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.127242 (+ 1241us) server_negotiation.cc:407] Connection header received
0620 12:22:08.127446 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.127452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.127496 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.127547 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.128641 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.129472 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.130693 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.131040 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.131117 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.132931 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.132935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.132937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.132962 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.135031 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.137515 (+ 2484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.137519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.137521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.137600 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.137878 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.137882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.137884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.138080 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.138187 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.138525 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.138637 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:22:08.143450 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.125488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.0.98.62:40307)
0620 12:22:08.125595 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.125600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.128796 (+ 3196us) server_negotiation.cc:407] Connection header received
0620 12:22:08.128961 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.128965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.129008 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.129090 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.130069 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.130831 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.133301 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.133636 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.133732 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.140093 (+ 6361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.140096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.140099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.140124 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.141710 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.142161 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.142165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.142167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.142243 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.142537 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.142540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.142541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.142728 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.142854 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.143177 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.143301 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:22:08.147455 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:08.126913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.0.98.60:45941)
0620 12:22:08.127018 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:08.127024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:08.132112 (+ 5088us) server_negotiation.cc:407] Connection header received
0620 12:22:08.132290 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:08.132293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:08.132336 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:08.132386 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:08.133346 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.134150 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.139182 (+ 5032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:08.139548 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:08.139619 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:08.143763 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:08.143768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:08.143770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:08.143802 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:08.145543 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.146042 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.146045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.146047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.146132 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:08.146427 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:08.146431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:08.146432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:08.146631 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:08.146763 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:08.147112 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 12:22:08.147250 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:22:08.250102 13952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:08.250308 13952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.251370 13952 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e768fdcc85a4afc9864ce6c1ff0b692 (127.0.98.61:44165), 156ec9e5caf04f6e97ee958e76f05ca3 (127.0.98.60:45941)
I20260620 12:22:08.252017 13825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ff863d658014193b5c6a1f482ac0330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" is_pre_election: true
I20260620 12:22:08.252334 13825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ff863d658014193b5c6a1f482ac0330 in term 0.
I20260620 12:22:08.252846 13727 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ff863d658014193b5c6a1f482ac0330, 9e768fdcc85a4afc9864ce6c1ff0b692; no voters:
I20260620 12:22:08.253193 13952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:22:08.253285 13952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:22:08.253327 13952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:08.254628 13952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.254920 13952 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [CANDIDATE]: Term 1 election: Requested vote from peers 9e768fdcc85a4afc9864ce6c1ff0b692 (127.0.98.61:44165), 156ec9e5caf04f6e97ee958e76f05ca3 (127.0.98.60:45941)
I20260620 12:22:08.255319 13825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ff863d658014193b5c6a1f482ac0330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692"
I20260620 12:22:08.255424 13825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:08.256507 13825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ff863d658014193b5c6a1f482ac0330 in term 1.
I20260620 12:22:08.256826 13727 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ff863d658014193b5c6a1f482ac0330, 9e768fdcc85a4afc9864ce6c1ff0b692; no voters:
I20260620 12:22:08.257438 13903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ff863d658014193b5c6a1f482ac0330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" is_pre_election: true
I20260620 12:22:08.257710 13903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ff863d658014193b5c6a1f482ac0330 in term 0.
I20260620 12:22:08.258462 13903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ff863d658014193b5c6a1f482ac0330" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156ec9e5caf04f6e97ee958e76f05ca3"
I20260620 12:22:08.258567 13903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:08.259692 13903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ff863d658014193b5c6a1f482ac0330 in term 1.
I20260620 12:22:08.260097 13952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:08.260345 13952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [term 1 LEADER]: Becoming Leader. State: Replica: 8ff863d658014193b5c6a1f482ac0330, State: Running, Role: LEADER
I20260620 12:22:08.260673 13952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } }
I20260620 12:22:08.262364 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff863d658014193b5c6a1f482ac0330. Latest consensus state: current_term: 1 leader_uuid: "8ff863d658014193b5c6a1f482ac0330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.262543 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:08.265336 13988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:08.266309 13988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:08.271720 13903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [term 1 FOLLOWER]: Refusing update from remote peer 8ff863d658014193b5c6a1f482ac0330: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:08.272337 13986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:08.275959 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff863d658014193b5c6a1f482ac0330. Latest consensus state: current_term: 1 leader_uuid: "8ff863d658014193b5c6a1f482ac0330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.276083 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:22:08.277033 13825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [term 1 FOLLOWER]: Refusing update from remote peer 8ff863d658014193b5c6a1f482ac0330: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:08.277637 13986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:08.281526 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff863d658014193b5c6a1f482ac0330. Latest consensus state: current_term: 1 leader_uuid: "8ff863d658014193b5c6a1f482ac0330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.281651 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:22:08.287312 13986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ff863d658014193b5c6a1f482ac0330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.287441 13986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:08.287669 13989 mvcc.cc:205] Tried to move back new op lower bound from 7298900493392846848 to 7298900493358829568. Current Snapshot: MvccSnapshot[applied={T|T < 7298900493392846848}]
I20260620 12:22:08.288533 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ff863d658014193b5c6a1f482ac0330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.288642 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:22:08.289496 13992 mvcc.cc:205] Tried to move back new op lower bound from 7298900493392846848 to 7298900493358829568. Current Snapshot: MvccSnapshot[applied={T|T < 7298900493392846848}]
I20260620 12:22:08.289682 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ff863d658014193b5c6a1f482ac0330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.289846 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330 [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:08.290625 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ff863d658014193b5c6a1f482ac0330" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff863d658014193b5c6a1f482ac0330" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } peers { permanent_uuid: "9e768fdcc85a4afc9864ce6c1ff0b692" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44165 } } peers { permanent_uuid: "156ec9e5caf04f6e97ee958e76f05ca3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45941 } } }
I20260620 12:22:08.290742 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:22:08.292295 13990 mvcc.cc:205] Tried to move back new op lower bound from 7298900493392846848 to 7298900493358829568. Current Snapshot: MvccSnapshot[applied={T|T < 7298900493392846848}]
I20260620 12:22:08.293368 13988 catalog_manager.cc:1347] Generated new cluster ID: 6c0e337b8e3e467ab704447f0e68fa13
I20260620 12:22:08.293428 13988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:22:08.314882 13988 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:22:08.316190 13988 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:22:08.327348 13988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff863d658014193b5c6a1f482ac0330: Generated new TSK 0
I20260620 12:22:08.328701 13988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:22:09.114030 13951 catalog_manager.cc:1259] Loaded cluster ID: 6c0e337b8e3e467ab704447f0e68fa13
I20260620 12:22:09.114105 13951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692: loading cluster ID for follower catalog manager: success
I20260620 12:22:09.115978 13951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692: acquiring CA information for follower catalog manager: success
I20260620 12:22:09.117234 13951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e768fdcc85a4afc9864ce6c1ff0b692: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:22:09.129294 13981 catalog_manager.cc:1259] Loaded cluster ID: 6c0e337b8e3e467ab704447f0e68fa13
I20260620 12:22:09.129359 13981 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3: loading cluster ID for follower catalog manager: success
I20260620 12:22:09.130949 13981 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3: acquiring CA information for follower catalog manager: success
I20260620 12:22:09.132066 13981 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 156ec9e5caf04f6e97ee958e76f05ca3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:22:09.382929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 13710
I20260620 12:22:09.389719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 13785
I20260620 12:22:09.395581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 13860
2026-06-20T12:22:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3124 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:22:09 dist-test-slave-9b3m krb5kdc[14003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:22:09 dist-test-slave-9b3m krb5kdc[14003](info): set up 2 sockets
Jun 20 12:22:09 dist-test-slave-9b3m krb5kdc[14003](info): commencing operation
krb5kdc: starting...
W20260620 12:22:10.449723 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:22:10 dist-test-slave-9b3m krb5kdc[14003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:22:10.568459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:10.673184 14016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:10.673470 14016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:10.673540 14016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:10.677429 14016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:10.677515 14016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:10.677554 14016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:10.677609 14016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:10.677644 14016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:10.682631 14016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:10.683571 14016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:10.684504 14016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:10.692741 14021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:10.692853 14016 server_base.cc:1034] running on GCE node
W20260620 12:22:10.692775 14022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:10.692818 14024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:10.693563 14016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:10.694520 14016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:10.695715 14016 hybrid_clock.cc:648] HybridClock initialized: now 1781958130695702 us; error 33 us; skew 500 ppm
Jun 20 12:22:10 dist-test-slave-9b3m krb5kdc[14003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:10.698321 14016 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:10.699110 14016 webserver.cc:460] Webserver started at http://127.0.98.62:41429/ using document root <none> and password file <none>
I20260620 12:22:10.699645 14016 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:10.699735 14016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:10.699950 14016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:10.701992 14016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "15ce54cbd712451da5ae9389e1863702"
format_stamp: "Formatted at 2026-06-20 12:22:10 on dist-test-slave-9b3m"
server_key: "5873a6bf8b241010c1b3bc3e5efdb757"
server_key_iv: "1cdd0056e4e01c402c9d7e2a6b2b700c"
server_key_version: "encryptionkey@0"
I20260620 12:22:10.702549 14016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "15ce54cbd712451da5ae9389e1863702"
format_stamp: "Formatted at 2026-06-20 12:22:10 on dist-test-slave-9b3m"
server_key: "5873a6bf8b241010c1b3bc3e5efdb757"
server_key_iv: "1cdd0056e4e01c402c9d7e2a6b2b700c"
server_key_version: "encryptionkey@0"
I20260620 12:22:10.706086 14016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:22:10.708318 14031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:10.709506 14016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:10.709645 14016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "15ce54cbd712451da5ae9389e1863702"
format_stamp: "Formatted at 2026-06-20 12:22:10 on dist-test-slave-9b3m"
server_key: "5873a6bf8b241010c1b3bc3e5efdb757"
server_key_iv: "1cdd0056e4e01c402c9d7e2a6b2b700c"
server_key_version: "encryptionkey@0"
I20260620 12:22:10.709750 14016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:10.729321 14016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:10.731943 14016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:10.732144 14016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:10.738858 14016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43997
I20260620 12:22:10.738890 14083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43997 every 8 connection(s)
I20260620 12:22:10.739778 14016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:10.742398 14084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:10.743923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 14016
I20260620 12:22:10.744024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:22:10.744308 392 external_mini_cluster.cc:1428] Setting key 72598c95a10e3a3aeb99961474d79d7d
I20260620 12:22:10.748093 14084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702: Bootstrap starting.
I20260620 12:22:10.750288 14084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:10.751020 14084 log.cc:824] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:22:10 dist-test-slave-9b3m krb5kdc[14003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:10.752975 14084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702: No bootstrap required, opened a new log
I20260620 12:22:10.756479 14084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ce54cbd712451da5ae9389e1863702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260620 12:22:10.756721 14084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:10.756814 14084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ce54cbd712451da5ae9389e1863702, State: Initialized, Role: FOLLOWER
I20260620 12:22:10.757274 14084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ce54cbd712451da5ae9389e1863702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260620 12:22:10.757403 14084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:22:10.757498 14084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:22:10.757629 14084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:10.758002 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:10.745627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.0.98.62:43997)
0620 12:22:10.746021 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:10.746035 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:10.746070 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:22:10.746677 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:10.746694 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:10.746991 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:10.747309 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:10.748078 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:10.749127 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:10.749978 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:10.750494 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:10.750675 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:10.753385 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:10.753391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:10.753402 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:10.753427 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:10.755058 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:10.755498 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:10.755504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:10.755516 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:10.755628 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:10.755907 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:10.755915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:10.755917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:10.756288 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:10.756505 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:10.756764 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 12:22:10.757106 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":125,"threads_started":1}
I20260620 12:22:10.758627 14084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ce54cbd712451da5ae9389e1863702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260620 12:22:10.758953 14084 leader_election.cc:302] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15ce54cbd712451da5ae9389e1863702; no voters:
I20260620 12:22:10.759291 14084 leader_election.cc:288] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:22:10.759343 14089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:10.759521 14089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [term 1 LEADER]: Becoming Leader. State: Replica: 15ce54cbd712451da5ae9389e1863702, State: Running, Role: LEADER
I20260620 12:22:10.760157 14089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ce54cbd712451da5ae9389e1863702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260620 12:22:10.760499 14084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:10.760949 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:22:10.761746 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15ce54cbd712451da5ae9389e1863702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ce54cbd712451da5ae9389e1863702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } } }
I20260620 12:22:10.761855 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:10.762208 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15ce54cbd712451da5ae9389e1863702. Latest consensus state: current_term: 1 leader_uuid: "15ce54cbd712451da5ae9389e1863702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ce54cbd712451da5ae9389e1863702" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } } }
I20260620 12:22:10.762316 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702 [sys.catalog]: This master's current role is: LEADER
W20260620 12:22:10.765038 14104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:22:10.765161 14104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:22:10.765223 14098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:10.765887 14098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:10.771185 14098 catalog_manager.cc:1347] Generated new cluster ID: 2ee34b1c137c44698830e2a7af14e33a
I20260620 12:22:10.771258 14098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:22:10.778927 14098 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:22:10.779744 14098 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:22:10.783691 14098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15ce54cbd712451da5ae9389e1863702: Generated new TSK 0
I20260620 12:22:10.784255 14098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:22:10.816776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 14016
2026-06-20T12:22:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1419 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:22:10 dist-test-slave-9b3m krb5kdc[14111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:22:10 dist-test-slave-9b3m krb5kdc[14111](info): set up 2 sockets
Jun 20 12:22:10 dist-test-slave-9b3m krb5kdc[14111](info): commencing operation
krb5kdc: starting...
W20260620 12:22:11.925333 392 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:22:12.044556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37673
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:12.157750 14124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:12.158007 14124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:12.158046 14124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:12.161900 14124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:12.161947 14124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:12.161965 14124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:12.161981 14124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:12.161996 14124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:12.167027 14124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:12.167982 14124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:12.168900 14124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:12.177314 14129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:12.177290 14130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:12.177430 14132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:12.177632 14124 server_base.cc:1034] running on GCE node
I20260620 12:22:12.178212 14124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:12.179286 14124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:12.180509 14124 hybrid_clock.cc:648] HybridClock initialized: now 1781958132180465 us; error 37 us; skew 500 ppm
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:12.187865 14124 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:12.188705 14124 webserver.cc:460] Webserver started at http://127.0.98.62:41383/ using document root <none> and password file <none>
I20260620 12:22:12.189316 14124 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:12.189409 14124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:12.189661 14124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:12.191689 14124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "9dec88912b474ba98c9501fb6b82345b"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "ce3fbd9747092fcc754630442fac6978"
server_key_iv: "148f1b8ea2c76ec2cf59e20b6d11c4de"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.192224 14124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "9dec88912b474ba98c9501fb6b82345b"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "ce3fbd9747092fcc754630442fac6978"
server_key_iv: "148f1b8ea2c76ec2cf59e20b6d11c4de"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.195753 14124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:22:12.198288 14139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.199640 14124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:22:12.199770 14124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "9dec88912b474ba98c9501fb6b82345b"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "ce3fbd9747092fcc754630442fac6978"
server_key_iv: "148f1b8ea2c76ec2cf59e20b6d11c4de"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.199880 14124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:12.214671 14124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:12.217690 14124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:12.217890 14124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:12.225490 14124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37673
I20260620 12:22:12.225795 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37673 every 8 connection(s)
I20260620 12:22:12.226432 14124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:12.229143 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:12.230715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 14124
I20260620 12:22:12.230870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:22:12.231177 392 external_mini_cluster.cc:1428] Setting key e41597bd6d2305e65f6c1a6e05864352
I20260620 12:22:12.235509 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Bootstrap starting.
I20260620 12:22:12.237725 14192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:12.238565 14192 log.cc:824] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:12.240749 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: No bootstrap required, opened a new log
I20260620 12:22:12.244765 14192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:12.244961 14192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:12.245028 14192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec88912b474ba98c9501fb6b82345b, State: Initialized, Role: FOLLOWER
I20260620 12:22:12.245195 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.232401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.0.98.62:37673)
0620 12:22:12.232789 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:12.232805 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:12.232842 (+ 37us) server_negotiation.cc:407] Connection header received
0620 12:22:12.233565 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:12.233587 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:12.233932 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:12.234274 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:12.235067 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.236130 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.237007 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.237527 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.237699 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.240616 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:12.240621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:12.240633 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:12.240657 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:12.242325 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.242854 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.242859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.242863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.242966 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.243258 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.243260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.243262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.243645 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:12.243786 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:12.244076 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 12:22:12.244352 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260620 12:22:12.245561 14192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:12.245719 14192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:22:12.245782 14192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:22:12.245872 14192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:12.246742 14192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:12.247061 14192 leader_election.cc:302] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dec88912b474ba98c9501fb6b82345b; no voters:
I20260620 12:22:12.247402 14192 leader_election.cc:288] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:22:12.247465 14197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:12.247659 14197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 1 LEADER]: Becoming Leader. State: Replica: 9dec88912b474ba98c9501fb6b82345b, State: Running, Role: LEADER
I20260620 12:22:12.247938 14197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:12.248593 14192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:12.249585 14198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dec88912b474ba98c9501fb6b82345b. Latest consensus state: current_term: 1 leader_uuid: "9dec88912b474ba98c9501fb6b82345b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } } }
I20260620 12:22:12.249624 14199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dec88912b474ba98c9501fb6b82345b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } } }
I20260620 12:22:12.249696 14198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:12.249724 14199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:12.250043 14206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:12.253067 14206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:12.258399 14206 catalog_manager.cc:1347] Generated new cluster ID: 3fd689694c6f4219a7841cb3f0a20cbe
I20260620 12:22:12.258471 14206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:22:12.267140 14206 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:22:12.268000 14206 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:22:12.274111 14206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Generated new TSK 0
I20260620 12:22:12.274744 14206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:22:12.305572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:12.423750 14218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:12.424010 14218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:12.424053 14218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:12.427911 14218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:12.427968 14218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:12.428025 14218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:22:12.433135 14218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:12.434080 14218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:12.435055 14218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:12.444496 14224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:12.444491 14226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:12.444679 14223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:12.444775 14218 server_base.cc:1034] running on GCE node
I20260620 12:22:12.445261 14218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:12.445850 14218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:12.447039 14218 hybrid_clock.cc:648] HybridClock initialized: now 1781958132447018 us; error 33 us; skew 500 ppm
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:12.449798 14218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:22:12.450737 14218 webserver.cc:460] Webserver started at http://127.0.98.1:36091/ using document root <none> and password file <none>
I20260620 12:22:12.451293 14218 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:12.451383 14218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:12.451583 14218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:12.453624 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea44ffd0be8f403bbf80850d5e69e175"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "dcd2195223eb4ea176bd1d8ca848bbf3"
server_key_iv: "298297435c657e3ab66f9a9678d5b3d8"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.454173 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea44ffd0be8f403bbf80850d5e69e175"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "dcd2195223eb4ea176bd1d8ca848bbf3"
server_key_iv: "298297435c657e3ab66f9a9678d5b3d8"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.457705 14218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:22:12.460039 14233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.461282 14218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:12.461400 14218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "ea44ffd0be8f403bbf80850d5e69e175"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "dcd2195223eb4ea176bd1d8ca848bbf3"
server_key_iv: "298297435c657e3ab66f9a9678d5b3d8"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.461521 14218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:12.483867 14218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:12.486617 14218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:12.486825 14218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:12.487360 14218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:12.488119 14218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:12.488188 14218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.488255 14218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:12.488302 14218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.496986 14218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37991
I20260620 12:22:12.497020 14346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37991 every 8 connection(s)
I20260620 12:22:12.497869 14218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:22:12.502235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 14218
I20260620 12:22:12.502408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:22:12.502728 392 external_mini_cluster.cc:1428] Setting key f6f8337809c1648b5c9737a6826291d9
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:22:12.512567 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.499696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38523 (local address 127.0.98.62:37673)
0620 12:22:12.499848 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:12.499851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:12.500617 (+ 766us) server_negotiation.cc:407] Connection header received
0620 12:22:12.501804 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:12.501806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:12.501855 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:12.501949 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:12.503217 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.503986 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.505181 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.505427 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.505541 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.508995 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:12.508998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:12.509001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:12.509027 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:12.510548 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.511064 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.511069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.511071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.511157 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.511511 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.511514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.511516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.511702 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:12.511807 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:12.512313 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 12:22:12.512443 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:22:12.513628 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.500040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:38523)
0620 12:22:12.500444 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:12.500507 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:12.501610 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:12.502091 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:12.502098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:12.502544 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:12.503062 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.503076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.504095 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.504098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.504982 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.504991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.505575 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.505579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.505753 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.506355 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:12.506371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:12.508825 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:12.510684 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:12.510690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:12.510704 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:12.510962 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:12.511250 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:12.511254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:12.511256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:12.511398 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:12.511806 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:12.511814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:12.512069 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 12:22:12.512274 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 12:22:12.512597 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:22:12.514925 14347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260620 12:22:12.515175 14347 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:12.515722 14347 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260620 12:22:12.517349 14156 ts_manager.cc:194] Registered new tserver with Master: ea44ffd0be8f403bbf80850d5e69e175 (127.0.98.1:37991)
I20260620 12:22:12.519060 14156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:22:12.529690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:12.636442 14352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:12.636718 14352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:12.636776 14352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:12.640681 14352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:12.640729 14352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:12.640813 14352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:22:12.645881 14352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:12.646847 14352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:12.647825 14352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:12.657099 14360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:12.657074 14358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:12.657109 14357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:12.657310 14352 server_base.cc:1034] running on GCE node
I20260620 12:22:12.657775 14352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:12.658429 14352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:12.659610 14352 hybrid_clock.cc:648] HybridClock initialized: now 1781958132659601 us; error 31 us; skew 500 ppm
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:12.662256 14352 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:22:12.663162 14352 webserver.cc:460] Webserver started at http://127.0.98.2:42161/ using document root <none> and password file <none>
I20260620 12:22:12.663709 14352 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:12.663798 14352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:12.664024 14352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:12.666276 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa25929be13c48769952c5725dfb5c9f"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "1053c75a19634d42c605b269d4649c06"
server_key_iv: "a447ed7106ad99575344f86cf880d697"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.666838 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa25929be13c48769952c5725dfb5c9f"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "1053c75a19634d42c605b269d4649c06"
server_key_iv: "a447ed7106ad99575344f86cf880d697"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.670332 14352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:22:12.672600 14367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.673795 14352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:12.673925 14352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "fa25929be13c48769952c5725dfb5c9f"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "1053c75a19634d42c605b269d4649c06"
server_key_iv: "a447ed7106ad99575344f86cf880d697"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.674028 14352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:12.690068 14352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:12.692780 14352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:12.692977 14352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:12.693549 14352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:12.694308 14352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:12.694379 14352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.694456 14352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:12.694499 14352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.703362 14352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44877
I20260620 12:22:12.703428 14480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44877 every 8 connection(s)
I20260620 12:22:12.704293 14352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:22:12.705137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 14352
I20260620 12:22:12.705260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:22:12.705570 392 external_mini_cluster.cc:1428] Setting key 3a79ed7033496768ec2f9843fe4eb62c
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:12.722971 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.706187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43385 (local address 127.0.98.62:37673)
0620 12:22:12.706317 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:12.706321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:12.708694 (+ 2373us) server_negotiation.cc:407] Connection header received
0620 12:22:12.710008 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:12.710011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:12.710065 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:12.710152 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:12.711376 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.712269 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.713523 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.713928 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.714049 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.719045 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:12.719049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:12.719055 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:12.719092 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:12.720672 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.721362 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.721366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.721375 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.721457 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.721835 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.721843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.721845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.722027 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:12.722140 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:12.722659 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 12:22:12.722786 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:22:12.723829 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.706589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:43385)
0620 12:22:12.708550 (+ 1961us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:12.708600 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:12.709804 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:12.710330 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:12.710337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:12.710689 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:12.711198 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.711222 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.712454 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.712457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.713397 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.713407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.714070 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.714072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.714204 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.714803 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:12.714819 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:12.718868 (+ 4049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:12.720794 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:12.720799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:12.720811 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:12.721187 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:12.721563 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:12.721569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:12.721571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:12.721711 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:12.722151 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:12.722160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:12.722409 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 12:22:12.722598 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 12:22:12.722937 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
I20260620 12:22:12.725143 14481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260620 12:22:12.725401 14481 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:12.725899 14481 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260620 12:22:12.726949 14156 ts_manager.cc:194] Registered new tserver with Master: fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877)
I20260620 12:22:12.727679 14156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:22:12.734050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:12.842428 14486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:12.842646 14486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:12.842686 14486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:12.846547 14486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:12.846593 14486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:12.846647 14486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:22:12.851824 14486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:12.852711 14486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:12.853645 14486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:12.862675 14491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:12.862702 14492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:12.862704 14494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:12.862818 14486 server_base.cc:1034] running on GCE node
I20260620 12:22:12.863345 14486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:12.863992 14486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:12.865170 14486 hybrid_clock.cc:648] HybridClock initialized: now 1781958132865161 us; error 28 us; skew 500 ppm
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:12.867893 14486 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:22:12.868685 14486 webserver.cc:460] Webserver started at http://127.0.98.3:36409/ using document root <none> and password file <none>
I20260620 12:22:12.869294 14486 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:12.869381 14486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:12.869606 14486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:12.871505 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "8fd733c5e4274df287305058cb01f75e"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "5d064318077892fdfb927494c2990e80"
server_key_iv: "4e3e551a53b8c6cca98b4ea31604becb"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.872049 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8fd733c5e4274df287305058cb01f75e"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "5d064318077892fdfb927494c2990e80"
server_key_iv: "4e3e551a53b8c6cca98b4ea31604becb"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.875751 14486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:22:12.878959 14501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.880733 14486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:22:12.880843 14486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "8fd733c5e4274df287305058cb01f75e"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "5d064318077892fdfb927494c2990e80"
server_key_iv: "4e3e551a53b8c6cca98b4ea31604becb"
server_key_version: "encryptionkey@0"
I20260620 12:22:12.880963 14486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:12.898463 14486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:12.901510 14486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:12.901706 14486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:12.902283 14486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:12.903090 14486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:12.903160 14486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.903221 14486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:12.903255 14486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:12.911815 14486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37773
I20260620 12:22:12.911831 14614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37773 every 8 connection(s)
I20260620 12:22:12.912679 14486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:22:12.919827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 14486
I20260620 12:22:12.919929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:22:12.920318 392 external_mini_cluster.cc:1428] Setting key 772c69322d52b8d7d1b85ebee8b324aa
Jun 20 12:22:12 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:12.926349 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.914445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33203 (local address 127.0.98.62:37673)
0620 12:22:12.914584 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:12.914588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:12.915379 (+ 791us) server_negotiation.cc:407] Connection header received
0620 12:22:12.916640 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:12.916643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:12.916704 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:12.916800 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:12.917998 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.918535 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.919659 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.919903 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.920035 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.923017 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:12.923021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:12.923023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:12.923049 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:12.924346 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.924846 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.924851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.924853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.924938 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.925296 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.925301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.925302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.925490 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:12.925597 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:12.926072 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 12:22:12.926187 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:22:12.927083 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.914829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:33203)
0620 12:22:12.915242 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:12.915281 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:12.916429 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:12.916975 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:12.916982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:12.917384 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:12.917844 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.917857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.918657 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.918663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.919534 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.919542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.920114 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.920119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.920320 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.920977 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:12.920999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:12.922852 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:12.924475 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:12.924480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:12.924495 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:12.924743 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:12.925030 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:12.925035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:12.925037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:12.925199 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:12.925597 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:12.925605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:12.925837 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 12:22:12.926018 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 12:22:12.926320 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260620 12:22:12.928079 14615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260620 12:22:12.928274 14615 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:12.928671 14615 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260620 12:22:12.929574 14156 ts_manager.cc:194] Registered new tserver with Master: 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773)
I20260620 12:22:12.930074 14156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33203
I20260620 12:22:12.934270 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:22:12.943007 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.936013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.98.62:37673)
0620 12:22:12.936150 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:12.936156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:12.936335 (+ 179us) server_negotiation.cc:407] Connection header received
0620 12:22:12.936487 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:12.936490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:12.936529 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:12.936617 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:12.937324 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.937824 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.938640 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.938871 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.938979 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.940188 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:12.940192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:12.940195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:12.940217 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:12.941421 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.941819 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.941826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.941827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.941906 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:12.942168 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:12.942172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:12.942174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:12.942372 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:12.942514 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:12.942743 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 12:22:12.942850 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:22:12.945554 14156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:22:12.948405 14156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:22:12.962563 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.957249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.98.3:37773)
0620 12:22:12.957683 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:12.957695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:12.957864 (+ 169us) server_negotiation.cc:407] Connection header received
0620 12:22:12.957999 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:12.958006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:12.958193 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:12.958328 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:12.958765 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.959475 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.961209 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.961844 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.961931 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.962042 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:12.962298 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 12:22:12.962402 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":146,"threads_started":1}
I20260620 12:22:12.962553 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.956889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44877 (local address 127.0.0.1:60216)
0620 12:22:12.957809 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:12.957822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:12.957897 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:12.958547 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:12.958551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:12.958573 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:12.958655 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.958661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.960009 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.960012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.961320 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.961330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.962158 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.962161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.962222 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.962271 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 12:22:12.962343 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 12:22:12.962408 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":115,"thread_start_us":86,"threads_started":1}
I20260620 12:22:12.962555 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.956666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37773 (local address 127.0.0.1:40304)
0620 12:22:12.957757 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:12.957769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:12.957881 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:12.958503 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:12.958506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:12.958523 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:12.958634 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.958644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.959657 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.959662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.961088 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.961099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.961970 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.961974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.962098 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.962150 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 12:22:12.962254 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 12:22:12.962372 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":128,"thread_start_us":113,"threads_started":1}
I20260620 12:22:12.963178 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.957322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.0.98.2:44877)
0620 12:22:12.958127 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:12.958138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:12.958156 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:22:12.958225 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:12.958229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:12.958389 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:12.958521 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:12.958781 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.959856 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.961447 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.962029 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.962135 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.962272 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:12.962934 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 12:22:12.963040 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":121,"threads_started":1}
I20260620 12:22:12.963955 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.956509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37991 (local address 127.0.0.1:48348)
0620 12:22:12.957288 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:12.957336 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:12.957495 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:12.958234 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:12.958239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:12.958260 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:12.958400 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.958407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.959903 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.959909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.961336 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:12.961343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.963558 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.963562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:12.963658 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.963680 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:22:12.963757 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 12:22:12.963822 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":114,"thread_start_us":65,"threads_started":1}
I20260620 12:22:12.964062 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:12.956668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.1:37991)
0620 12:22:12.957770 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:12.957781 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:12.957803 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:22:12.957883 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:12.957888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:12.958085 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:12.958245 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:12.958516 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.959752 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.962332 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:12.963438 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:12.963544 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:12.963668 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:12.963796 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 12:22:12.963930 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":109,"threads_started":1}
I20260620 12:22:12.965847 14415 tablet_service.cc:1467] Processing CreateTablet for tablet cec695287da642aabe2d6fc30423f6a5 (DEFAULT_TABLE table=test-table [id=f49cabad46d843e1b789322bfa95dae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:12.965947 14549 tablet_service.cc:1467] Processing CreateTablet for tablet cec695287da642aabe2d6fc30423f6a5 (DEFAULT_TABLE table=test-table [id=f49cabad46d843e1b789322bfa95dae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:12.966867 14415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec695287da642aabe2d6fc30423f6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:12.966920 14549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec695287da642aabe2d6fc30423f6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:12.966774 14281 tablet_service.cc:1467] Processing CreateTablet for tablet cec695287da642aabe2d6fc30423f6a5 (DEFAULT_TABLE table=test-table [id=f49cabad46d843e1b789322bfa95dae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:12.967660 14281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec695287da642aabe2d6fc30423f6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:12.973480 14632 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Bootstrap starting.
I20260620 12:22:12.974411 14634 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Bootstrap starting.
I20260620 12:22:12.975863 14632 tablet_bootstrap.cc:654] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:12.976053 14634 tablet_bootstrap.cc:654] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:12.976629 14632 log.cc:824] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:12.977149 14634 log.cc:824] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:12.977373 14633 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Bootstrap starting.
I20260620 12:22:12.978767 14632 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: No bootstrap required, opened a new log
I20260620 12:22:12.978966 14632 ts_tablet_manager.cc:1397] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:22:12.979334 14634 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: No bootstrap required, opened a new log
I20260620 12:22:12.979516 14634 ts_tablet_manager.cc:1397] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:22:12.979662 14633 tablet_bootstrap.cc:654] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:12.980335 14633 log.cc:824] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:12.981873 14633 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: No bootstrap required, opened a new log
I20260620 12:22:12.982075 14633 ts_tablet_manager.cc:1397] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:22:12.982839 14632 raft_consensus.cc:348] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:12.983120 14632 raft_consensus.cc:374] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:12.983170 14632 raft_consensus.cc:729] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa25929be13c48769952c5725dfb5c9f, State: Initialized, Role: FOLLOWER
I20260620 12:22:12.983189 14634 raft_consensus.cc:348] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:12.983398 14634 raft_consensus.cc:374] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:12.983465 14634 raft_consensus.cc:729] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea44ffd0be8f403bbf80850d5e69e175, State: Initialized, Role: FOLLOWER
I20260620 12:22:12.983644 14632 consensus_queue.cc:260] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:12.983879 14634 consensus_queue.cc:260] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:12.984511 14347 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
I20260620 12:22:12.984709 14634 ts_tablet_manager.cc:1428] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:22:12.985141 14481 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
I20260620 12:22:12.985248 14632 ts_tablet_manager.cc:1428] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:22:12.985558 14633 raft_consensus.cc:348] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:12.985816 14633 raft_consensus.cc:374] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:12.985877 14633 raft_consensus.cc:729] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd733c5e4274df287305058cb01f75e, State: Initialized, Role: FOLLOWER
I20260620 12:22:12.986474 14633 consensus_queue.cc:260] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:12.987061 14615 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
I20260620 12:22:12.987367 14633 ts_tablet_manager.cc:1428] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20260620 12:22:12.998992 14348 tablet.cc:2369] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:22:13.163782 14616 tablet.cc:2369] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:22:13.205451 14482 tablet.cc:2369] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:22:13.217832 14639 raft_consensus.cc:482] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:13.218103 14639 raft_consensus.cc:504] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:13.219233 14639 leader_election.cc:288] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773), fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877)
I20260620 12:22:13.223677 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.219532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37773 (local address 127.0.98.1:55645)
0620 12:22:13.220060 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:13.220076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:13.220193 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:13.220502 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:13.220505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:13.220541 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:13.220657 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.220664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.221513 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.221515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.222385 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.222393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.223298 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.223301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.223374 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.223395 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 12:22:13.223458 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:22:13.223511 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":58,"thread_start_us":66,"threads_started":1}
I20260620 12:22:13.223711 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.219760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55645 (local address 127.0.98.3:37773)
0620 12:22:13.219943 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:13.219950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:13.220174 (+ 224us) server_negotiation.cc:407] Connection header received
0620 12:22:13.220341 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:13.220344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:13.220390 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:13.220471 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:13.220799 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.221394 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.222514 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.223143 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.223253 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.223317 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:13.223502 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 12:22:13.223586 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:22:13.223894 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.219676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50761 (local address 127.0.98.2:44877)
0620 12:22:13.219836 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:13.219840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:13.220012 (+ 172us) server_negotiation.cc:407] Connection header received
0620 12:22:13.220152 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:13.220155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:13.220210 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:13.220296 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:13.220676 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.221352 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.222557 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.223143 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.223253 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.223317 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:13.223660 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 12:22:13.223770 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 12:22:13.223831 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.219532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44877 (local address 127.0.98.1:50761)
0620 12:22:13.219882 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:13.219897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:13.220039 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:13.220340 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:13.220343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:13.220361 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:13.220521 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.220530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.221471 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.221475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.222383 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.222392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.223399 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.223403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.223524 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.223546 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:22:13.223621 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:22:13.223687 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":37,"thread_start_us":102,"threads_started":1}
I20260620 12:22:13.224677 14569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec695287da642aabe2d6fc30423f6a5" candidate_uuid: "ea44ffd0be8f403bbf80850d5e69e175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd733c5e4274df287305058cb01f75e" is_pre_election: true
I20260620 12:22:13.224768 14435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec695287da642aabe2d6fc30423f6a5" candidate_uuid: "ea44ffd0be8f403bbf80850d5e69e175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa25929be13c48769952c5725dfb5c9f" is_pre_election: true
I20260620 12:22:13.225010 14569 raft_consensus.cc:2454] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea44ffd0be8f403bbf80850d5e69e175 in term 0.
I20260620 12:22:13.225106 14435 raft_consensus.cc:2454] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea44ffd0be8f403bbf80850d5e69e175 in term 0.
I20260620 12:22:13.225508 14237 leader_election.cc:302] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fd733c5e4274df287305058cb01f75e, ea44ffd0be8f403bbf80850d5e69e175; no voters:
I20260620 12:22:13.225768 14639 raft_consensus.cc:2781] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:22:13.225873 14639 raft_consensus.cc:482] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:22:13.225925 14639 raft_consensus.cc:3037] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:13.226955 14639 raft_consensus.cc:504] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:13.227320 14639 leader_election.cc:288] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 1 election: Requested vote from peers 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773), fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877)
I20260620 12:22:13.227625 14435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec695287da642aabe2d6fc30423f6a5" candidate_uuid: "ea44ffd0be8f403bbf80850d5e69e175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa25929be13c48769952c5725dfb5c9f"
I20260620 12:22:13.227622 14569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec695287da642aabe2d6fc30423f6a5" candidate_uuid: "ea44ffd0be8f403bbf80850d5e69e175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd733c5e4274df287305058cb01f75e"
I20260620 12:22:13.227753 14435 raft_consensus.cc:3037] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:13.227742 14569 raft_consensus.cc:3037] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:13.228978 14435 raft_consensus.cc:2454] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea44ffd0be8f403bbf80850d5e69e175 in term 1.
I20260620 12:22:13.228978 14569 raft_consensus.cc:2454] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea44ffd0be8f403bbf80850d5e69e175 in term 1.
I20260620 12:22:13.229287 14235 leader_election.cc:302] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea44ffd0be8f403bbf80850d5e69e175, fa25929be13c48769952c5725dfb5c9f; no voters:
I20260620 12:22:13.229454 14639 raft_consensus.cc:2781] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:13.229673 14639 raft_consensus.cc:686] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 LEADER]: Becoming Leader. State: Replica: ea44ffd0be8f403bbf80850d5e69e175, State: Running, Role: LEADER
I20260620 12:22:13.230010 14639 consensus_queue.cc:237] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:13.233315 14155 catalog_manager.cc:5495] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea44ffd0be8f403bbf80850d5e69e175 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ea44ffd0be8f403bbf80850d5e69e175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } health_report { overall_health: UNKNOWN } } }
I20260620 12:22:13.257428 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.254067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.0.98.1:37991)
0620 12:22:13.254183 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:13.254186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:13.254199 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:22:13.254318 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:13.254320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:13.254374 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:13.254471 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:22:13.254663 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.255149 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.255959 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.256153 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.256229 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.256606 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:22:13.257031 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:22:13.257232 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:13.257270 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 12:22:13.257321 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:22:13.263132 14435 raft_consensus.cc:1261] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 1 FOLLOWER]: Refusing update from remote peer ea44ffd0be8f403bbf80850d5e69e175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:13.263132 14569 raft_consensus.cc:1261] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 1 FOLLOWER]: Refusing update from remote peer ea44ffd0be8f403bbf80850d5e69e175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:13.263789 14639 consensus_queue.cc:1035] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:13.264020 14643 consensus_queue.cc:1035] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:13.275756 14647 mvcc.cc:205] Tried to move back new op lower bound from 7298900513838350336 to 7298900513712922624. Current Snapshot: MvccSnapshot[applied={T|T < 7298900513838350336 or (T in {7298900513838350336})}]
I20260620 12:22:13.279989 14155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:13.280177 14155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:13.282502 14155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Sending DeleteTablet for 3 replicas of tablet cec695287da642aabe2d6fc30423f6a5
I20260620 12:22:13.283231 14281 tablet_service.cc:1514] Processing DeleteTablet for tablet cec695287da642aabe2d6fc30423f6a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48348
I20260620 12:22:13.283255 14415 tablet_service.cc:1514] Processing DeleteTablet for tablet cec695287da642aabe2d6fc30423f6a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260620 12:22:13.283740 14657 tablet_replica.cc:330] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: stopping tablet replica
I20260620 12:22:13.283731 14549 tablet_service.cc:1514] Processing DeleteTablet for tablet cec695287da642aabe2d6fc30423f6a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40304
I20260620 12:22:13.284061 14657 raft_consensus.cc:2229] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:13.284194 14658 tablet_replica.cc:330] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: stopping tablet replica
I20260620 12:22:13.284399 14657 raft_consensus.cc:2258] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:13.284528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 14218
I20260620 12:22:13.284618 14658 raft_consensus.cc:2229] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:13.285445 14658 raft_consensus.cc:2258] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:13.287020 14658 ts_tablet_manager.cc:1905] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:13.287230 14657 ts_tablet_manager.cc:1905] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:13.289152 14658 ts_tablet_manager.cc:1918] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:13.289260 14658 log.cc:1196] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/cec695287da642aabe2d6fc30423f6a5
I20260620 12:22:13.289546 14658 ts_tablet_manager.cc:1939] T cec695287da642aabe2d6fc30423f6a5 P 8fd733c5e4274df287305058cb01f75e: Deleting consensus metadata
I20260620 12:22:13.290620 14143 catalog_manager.cc:4841] TS 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773): tablet cec695287da642aabe2d6fc30423f6a5 (table test-table [id=f49cabad46d843e1b789322bfa95dae7]) successfully deleted
I20260620 12:22:13.290653 14657 ts_tablet_manager.cc:1918] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:13.290751 14657 log.cc:1196] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/cec695287da642aabe2d6fc30423f6a5
I20260620 12:22:13.291072 14657 ts_tablet_manager.cc:1939] T cec695287da642aabe2d6fc30423f6a5 P fa25929be13c48769952c5725dfb5c9f: Deleting consensus metadata
W20260620 12:22:13.292089 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37991 (error 108)
I20260620 12:22:13.292304 14141 catalog_manager.cc:4841] TS fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877): tablet cec695287da642aabe2d6fc30423f6a5 (table test-table [id=f49cabad46d843e1b789322bfa95dae7]) successfully deleted
I20260620 12:22:13.292934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 14352
I20260620 12:22:13.293157 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.292792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37991 (local address 127.0.0.1:48370)
0620 12:22:13.292944 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:13.293039 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 12:22:13.293315 14142 catalog_manager.cc:4584] TS ea44ffd0be8f403bbf80850d5e69e175 (127.0.98.1:37991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cec695287da642aabe2d6fc30423f6a5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37991: connect: Connection refused (error 111)
I20260620 12:22:13.298717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 14486
I20260620 12:22:13.304280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 14124
I20260620 12:22:13.311030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37673
--webserver_interface=127.0.98.62
--webserver_port=41383
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:13.417656 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:13.417908 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:13.417949 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:13.421944 14660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:13.421993 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:13.422011 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:13.422029 14660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:13.422096 14660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:13.427145 14660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:13.428141 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:13.429154 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:13.441978 14666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:13.441978 14669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:13.442015 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:13.442046 14660 server_base.cc:1034] running on GCE node
I20260620 12:22:13.442718 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:13.443794 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:13.444989 14660 hybrid_clock.cc:648] HybridClock initialized: now 1781958133444970 us; error 29 us; skew 500 ppm
Jun 20 12:22:13 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:13.447749 14660 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:13.448609 14660 webserver.cc:460] Webserver started at http://127.0.98.62:41383/ using document root <none> and password file <none>
I20260620 12:22:13.449208 14660 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:13.449283 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:13.452860 14660 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:13.454907 14676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:13.456095 14660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:13.456194 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "9dec88912b474ba98c9501fb6b82345b"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "ce3fbd9747092fcc754630442fac6978"
server_key_iv: "148f1b8ea2c76ec2cf59e20b6d11c4de"
server_key_version: "encryptionkey@0"
I20260620 12:22:13.456753 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:13.470894 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:13.473453 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:13.473603 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:13.480338 14660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37673
I20260620 12:22:13.480613 14728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37673 every 8 connection(s)
I20260620 12:22:13.481408 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:13.485523 14729 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:22:13.487056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 14660
I20260620 12:22:13.487269 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Bootstrap starting.
I20260620 12:22:13.487550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37991
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37673
--webserver_port=36091
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:13.504748 14729 log.cc:824] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:13.513592 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:22:13.513991 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Bootstrap complete.
I20260620 12:22:13.518132 14729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:13.518519 14729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec88912b474ba98c9501fb6b82345b, State: Initialized, Role: FOLLOWER
I20260620 12:22:13.518983 14729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:13.519091 14729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:22:13.519182 14729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:22:13.519286 14729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:22:13.521166 14729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:13.521490 14729 leader_election.cc:302] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dec88912b474ba98c9501fb6b82345b; no voters:
I20260620 12:22:13.521849 14729 leader_election.cc:288] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:22:13.521981 14733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:22:13.522224 14733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 2 LEADER]: Becoming Leader. State: Replica: 9dec88912b474ba98c9501fb6b82345b, State: Running, Role: LEADER
I20260620 12:22:13.522526 14733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:13.523010 14729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:13.523442 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dec88912b474ba98c9501fb6b82345b. Latest consensus state: current_term: 2 leader_uuid: "9dec88912b474ba98c9501fb6b82345b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } } }
I20260620 12:22:13.523553 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:13.523834 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9dec88912b474ba98c9501fb6b82345b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } } }
I20260620 12:22:13.523905 14737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:13.523937 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:13.528527 14737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:13.529191 14737 catalog_manager.cc:1259] Loaded cluster ID: 3fd689694c6f4219a7841cb3f0a20cbe
I20260620 12:22:13.529242 14737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:22:13.530608 14737 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 12:22:13.535275 14737 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 12:22:13.540311 14737 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:22:13.540982 14737 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Loaded TSK: 0
I20260620 12:22:13.542184 14737 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:22:13.603322 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:13.603559 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:13.603602 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:13.607630 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:13.607686 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:13.607743 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:22:13.612814 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36091
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:13.613790 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:13.614737 14731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:13.625052 14756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:13.625092 14759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:13.625286 14731 server_base.cc:1034] running on GCE node
W20260620 12:22:13.625483 14757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:13.625883 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:13.626545 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:13.627727 14731 hybrid_clock.cc:648] HybridClock initialized: now 1781958133627701 us; error 47 us; skew 500 ppm
Jun 20 12:22:13 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:13.630599 14731 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:22:13.631511 14731 webserver.cc:460] Webserver started at http://127.0.98.1:36091/ using document root <none> and password file <none>
I20260620 12:22:13.632040 14731 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:13.632141 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:13.636034 14731 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:13.638163 14766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:13.639398 14731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:22:13.639487 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "ea44ffd0be8f403bbf80850d5e69e175"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "dcd2195223eb4ea176bd1d8ca848bbf3"
server_key_iv: "298297435c657e3ab66f9a9678d5b3d8"
server_key_version: "encryptionkey@0"
I20260620 12:22:13.640027 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:13.667523 14731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:13.670432 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:13.670629 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:13.671212 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:13.672277 14773 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:22:13.676230 14731 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:22:13.676283 14731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 12:22:13.676355 14731 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:22:13.677989 14731 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:22:13.678031 14731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260620 12:22:13.678460 14773 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Bootstrap starting.
I20260620 12:22:13.688301 14731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37991
I20260620 12:22:13.688410 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37991 every 8 connection(s)
I20260620 12:22:13.689320 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:22:13.694341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 14731
I20260620 12:22:13.695051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44877
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37673
--webserver_port=42161
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:13.697671 14773 log.cc:824] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:22:13 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:13.705897 14773 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:22:13.706347 14773 tablet_bootstrap.cc:492] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Bootstrap complete.
I20260620 12:22:13.707590 14773 ts_tablet_manager.cc:1397] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.007s
I20260620 12:22:13.708658 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.691356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57017 (local address 127.0.98.62:37673)
0620 12:22:13.691688 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:13.691709 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:13.692050 (+ 341us) server_negotiation.cc:407] Connection header received
0620 12:22:13.693161 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:13.693173 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:13.693464 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:13.693784 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:13.695090 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.696247 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.698042 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.698893 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.699097 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.702400 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:13.702407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:13.702421 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:13.702452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:13.704710 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:13.705471 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:13.705477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:13.705482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:13.705627 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:13.706105 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:13.706111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:13.706114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:13.706576 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:13.706724 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:13.707465 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 12:22:13.707779 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
I20260620 12:22:13.709100 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.691388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:57017)
0620 12:22:13.691892 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:13.691931 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:13.692703 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:13.693840 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:13.693848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:13.694250 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:13.694912 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.694926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.696391 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.696402 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.697882 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.697894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.699058 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.699062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.699244 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.699957 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:13.699976 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:13.702168 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:13.704851 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:13.704862 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:13.704878 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:13.705342 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:13.705760 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:13.705767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:13.705771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:13.705980 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:13.706723 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:13.706733 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:13.707093 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 12:22:13.707323 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 12:22:13.707839 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":165,"threads_started":1}
I20260620 12:22:13.710525 14773 raft_consensus.cc:348] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:13.711030 14773 raft_consensus.cc:729] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea44ffd0be8f403bbf80850d5e69e175, State: Initialized, Role: FOLLOWER
I20260620 12:22:13.711515 14773 consensus_queue.cc:260] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } }
I20260620 12:22:13.711697 14881 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260620 12:22:13.711925 14881 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:13.712437 14773 ts_tablet_manager.cc:1428] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260620 12:22:13.712463 14881 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260620 12:22:13.715204 14693 ts_manager.cc:194] Registered new tserver with Master: ea44ffd0be8f403bbf80850d5e69e175 (127.0.98.1:37991)
I20260620 12:22:13.719444 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57017
I20260620 12:22:13.721837 14881 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
Jun 20 12:22:13 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:22:13.729621 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.718662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37991 (local address 127.0.0.1:48374)
0620 12:22:13.719314 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:13.719349 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:13.719503 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:13.719997 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:13.720087 (+ 90us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:13.720492 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:13.720638 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.720646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.722246 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.722252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.722979 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.722989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.723718 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.723733 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.723847 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.724435 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:13.724445 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:13.726122 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:13.727926 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:13.727936 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:13.727941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:13.728160 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:13.728457 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:13.728461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:13.728463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:13.728545 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:13.728976 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:13.728982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:13.729115 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 12:22:13.729258 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 12:22:13.729449 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":116,"threads_started":1}
I20260620 12:22:13.729621 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.718704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.0.98.1:37991)
0620 12:22:13.719058 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:13.719069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:13.719440 (+ 371us) server_negotiation.cc:407] Connection header received
0620 12:22:13.719693 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:13.719698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:13.719843 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:13.719997 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:13.720768 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.722124 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.723098 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.723617 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.723717 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.726251 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:13.726260 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:13.726272 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:13.726302 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:13.727801 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:13.728272 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:13.728278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:13.728280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:13.728363 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:13.728631 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:13.728636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:13.728638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:13.728865 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:13.729001 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:13.729335 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 12:22:13.729483 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":86,"threads_started":1}
I20260620 12:22:13.730515 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet cec695287da642aabe2d6fc30423f6a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48374
I20260620 12:22:13.731318 14890 tablet_replica.cc:330] stopping tablet replica
I20260620 12:22:13.731629 14890 raft_consensus.cc:2229] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:13.731886 14890 raft_consensus.cc:2258] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:13.732968 14890 ts_tablet_manager.cc:1905] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:13.737722 14890 ts_tablet_manager.cc:1918] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:13.737843 14890 log.cc:1196] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/cec695287da642aabe2d6fc30423f6a5
I20260620 12:22:13.738193 14890 ts_tablet_manager.cc:1939] T cec695287da642aabe2d6fc30423f6a5 P ea44ffd0be8f403bbf80850d5e69e175: Deleting consensus metadata
I20260620 12:22:13.740084 14679 catalog_manager.cc:4841] TS ea44ffd0be8f403bbf80850d5e69e175 (127.0.98.1:37991): tablet cec695287da642aabe2d6fc30423f6a5 (table test-table [id=f49cabad46d843e1b789322bfa95dae7]) successfully deleted
W20260620 12:22:13.820452 14885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:13.820690 14885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:13.820766 14885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:13.824954 14885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:13.825042 14885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:13.825160 14885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:22:13.830341 14885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42161
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:13.831351 14885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:13.832351 14885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:13.842293 14896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:13.842291 14898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:13.842449 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:13.842537 14885 server_base.cc:1034] running on GCE node
I20260620 12:22:13.842962 14885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:13.843668 14885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:13.844879 14885 hybrid_clock.cc:648] HybridClock initialized: now 1781958133844846 us; error 47 us; skew 500 ppm
Jun 20 12:22:13 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:13.847738 14885 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:22:13.848596 14885 webserver.cc:460] Webserver started at http://127.0.98.2:42161/ using document root <none> and password file <none>
I20260620 12:22:13.849143 14885 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:13.849223 14885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:13.852917 14885 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:13.854904 14905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:13.856070 14885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:13.856160 14885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "fa25929be13c48769952c5725dfb5c9f"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "1053c75a19634d42c605b269d4649c06"
server_key_iv: "a447ed7106ad99575344f86cf880d697"
server_key_version: "encryptionkey@0"
I20260620 12:22:13.856705 14885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:13.871162 14885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:13.874425 14885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:13.874574 14885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:13.875069 14885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:13.875803 14885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:13.875847 14885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:13.875885 14885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:13.875906 14885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:13.884575 14885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44877
I20260620 12:22:13.884603 15018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44877 every 8 connection(s)
I20260620 12:22:13.885622 14885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:22:13.892791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 14885
I20260620 12:22:13.893294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37773
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37673
--webserver_port=36409
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:22:13 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:13.903599 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.887470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45081 (local address 127.0.98.62:37673)
0620 12:22:13.887745 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:13.887750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:13.888682 (+ 932us) server_negotiation.cc:407] Connection header received
0620 12:22:13.890007 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:13.890010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:13.890060 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:13.890145 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:13.891298 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.891877 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.893267 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.893695 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.893807 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.898520 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:13.898524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:13.898526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:13.898564 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:13.900693 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:13.901527 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:13.901531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:13.901532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:13.901625 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:13.902094 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:13.902099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:13.902101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:13.902395 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:13.902513 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:13.903271 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 12:22:13.903425 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 12:22:13.904569 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:13.888020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:45081)
0620 12:22:13.888520 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:13.888564 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:13.889787 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:13.890303 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:13.890309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:13.890645 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:13.891116 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.891129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.891986 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.891989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.893130 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:13.893141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:13.893912 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:13.893918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:13.894189 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:13.895046 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:13.895072 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:13.898324 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:13.900863 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:13.900869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:13.900886 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:13.901378 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:13.901733 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:13.901736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:13.901738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:13.901987 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:13.902522 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:13.902531 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:13.902934 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 12:22:13.903173 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 12:22:13.903516 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260620 12:22:13.905987 15019 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260620 12:22:13.906292 15019 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:13.906955 15019 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260620 12:22:13.908376 14693 ts_manager.cc:194] Registered new tserver with Master: fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877)
I20260620 12:22:13.909264 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45081
W20260620 12:22:14.016573 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:14.016858 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:14.016934 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:14.021097 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:14.021162 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:14.021245 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:22:14.026786 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36409
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:14.027779 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:14.028749 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:14.038574 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:14.038945 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:14.039022 15022 server_base.cc:1034] running on GCE node
W20260620 12:22:14.039047 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:14.039502 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:14.040158 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:14.041344 15022 hybrid_clock.cc:648] HybridClock initialized: now 1781958134041324 us; error 56 us; skew 500 ppm
Jun 20 12:22:14 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:14.045084 15022 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:22:14.046132 15022 webserver.cc:460] Webserver started at http://127.0.98.3:36409/ using document root <none> and password file <none>
I20260620 12:22:14.046854 15022 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:14.046967 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:14.051363 15022 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:22:14.053426 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:14.054600 15022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:14.054731 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "8fd733c5e4274df287305058cb01f75e"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "5d064318077892fdfb927494c2990e80"
server_key_iv: "4e3e551a53b8c6cca98b4ea31604becb"
server_key_version: "encryptionkey@0"
I20260620 12:22:14.055253 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:14.069448 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:14.072083 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:14.072275 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:14.072831 15022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:14.073662 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:14.073733 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:14.073802 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:14.073848 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:14.082621 15022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37773
I20260620 12:22:14.082640 15150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37773 every 8 connection(s)
I20260620 12:22:14.083575 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:22:14.091912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 15022
Jun 20 12:22:14 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:14.096793 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.085301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60135 (local address 127.0.98.62:37673)
0620 12:22:14.085432 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:14.085435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:14.086194 (+ 759us) server_negotiation.cc:407] Connection header received
0620 12:22:14.087460 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:14.087463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:14.087522 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:14.087623 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:14.088718 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.089301 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.090410 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.090630 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.090741 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.093319 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:14.093324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:14.093330 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:14.093362 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:14.094853 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:14.095310 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:14.095314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:14.095316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:14.095400 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:14.095698 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:14.095702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:14.095703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:14.095906 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:14.095979 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:14.096514 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 12:22:14.096626 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:22:14.097738 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.085641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:60135)
0620 12:22:14.086053 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.086092 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:14.087248 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:14.087802 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:14.087809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:14.088143 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:14.088581 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.088593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.089426 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.089429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.090284 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.090293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.090772 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.090776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.090944 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.091439 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:14.091453 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:14.093155 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:14.094960 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:14.094964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:14.094975 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:14.095214 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:14.095489 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:14.095491 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:14.095492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:14.095611 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:14.096000 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:14.096006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:14.096288 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 12:22:14.096473 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 12:22:14.096762 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260620 12:22:14.098901 15151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37673
I20260620 12:22:14.099136 15151 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:14.099679 15151 heartbeater.cc:511] Master 127.0.98.62:37673 requested a full tablet report, sending...
I20260620 12:22:14.100724 14693 ts_manager.cc:194] Registered new tserver with Master: 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773)
I20260620 12:22:14.100915 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.092603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.0.98.62:37673)
0620 12:22:14.092854 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:14.092858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:14.092933 (+ 75us) server_negotiation.cc:407] Connection header received
0620 12:22:14.093027 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:14.093033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:14.093101 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:14.093193 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:14.094128 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.094735 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.095788 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.096053 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.096150 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.097530 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:14.097534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:14.097537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:14.097560 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:14.099251 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:14.099619 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:14.099623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:14.099625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:14.099711 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:14.100004 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:14.100007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:14.100009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:14.100203 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:14.100477 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:14.100623 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 12:22:14.100740 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260620 12:22:14.101392 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60135
I20260620 12:22:14.103286 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:22:14.106937 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.103845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.98.62:37673)
0620 12:22:14.103960 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:14.103962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:14.104122 (+ 160us) server_negotiation.cc:407] Connection header received
0620 12:22:14.104231 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:14.104233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:14.104319 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:14.104390 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:22:14.104555 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.105008 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.105841 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.106047 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.106120 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.106351 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:22:14.106674 (+ 323us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:22:14.106753 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:14.106807 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 12:22:14.106851 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:22:14.107527 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:22:14.110724 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:22:14.118366 14815 tablet_service.cc:1467] Processing CreateTablet for tablet 09768033336e4c8bbe11197d49b588f3 (DEFAULT_TABLE table=test-table [id=bbcff8978fad4d08bebceb563f99566a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:14.119001 14815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09768033336e4c8bbe11197d49b588f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:14.121829 14773 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Bootstrap starting.
I20260620 12:22:14.122236 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.117286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37773 (local address 127.0.0.1:40310)
0620 12:22:14.117731 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.117742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:14.117819 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:14.118425 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:14.118428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:14.118449 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:14.118572 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.118579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.119705 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.119711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.120785 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.120794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.121776 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.121780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.121889 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.121938 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 12:22:14.122021 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 12:22:14.122089 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260620 12:22:14.122748 14773 tablet_bootstrap.cc:654] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:14.123036 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.116857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44877 (local address 127.0.0.1:60222)
0620 12:22:14.116988 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.117005 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:14.117097 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:14.118395 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:14.118400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:14.118418 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:14.118520 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.118527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.120160 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.120164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.121330 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.121340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.122556 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.122560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.122668 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.122707 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 12:22:14.122794 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:22:14.122913 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 12:22:14.123245 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.117230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.98.2:44877)
0620 12:22:14.117923 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:14.117937 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:14.117962 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:22:14.118037 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:14.118043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:14.118228 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:14.118663 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:14.118826 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.120016 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.121472 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.122447 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.122565 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.122700 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:14.122906 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 12:22:14.123008 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":149,"threads_started":1}
I20260620 12:22:14.122303 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.117313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.98.3:37773)
0620 12:22:14.117937 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:14.117948 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:14.117975 (+ 27us) server_negotiation.cc:407] Connection header received
0620 12:22:14.118034 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:14.118038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:14.118204 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:14.118465 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:14.118682 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.119579 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.120916 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.121642 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.121752 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.121890 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:14.122065 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 12:22:14.122161 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":148,"threads_started":1}
I20260620 12:22:14.124722 14773 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: No bootstrap required, opened a new log
I20260620 12:22:14.124830 14773 ts_tablet_manager.cc:1397] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260620 12:22:14.125272 14773 raft_consensus.cc:348] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.125396 14773 raft_consensus.cc:374] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:14.125442 14773 raft_consensus.cc:729] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea44ffd0be8f403bbf80850d5e69e175, State: Initialized, Role: FOLLOWER
I20260620 12:22:14.125612 14773 consensus_queue.cc:260] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.125957 14773 ts_tablet_manager.cc:1428] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 12:22:14.126524 15085 tablet_service.cc:1467] Processing CreateTablet for tablet 09768033336e4c8bbe11197d49b588f3 (DEFAULT_TABLE table=test-table [id=bbcff8978fad4d08bebceb563f99566a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:14.126524 14953 tablet_service.cc:1467] Processing CreateTablet for tablet 09768033336e4c8bbe11197d49b588f3 (DEFAULT_TABLE table=test-table [id=bbcff8978fad4d08bebceb563f99566a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:14.127619 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09768033336e4c8bbe11197d49b588f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:14.127625 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09768033336e4c8bbe11197d49b588f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:14.135869 15164 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: Bootstrap starting.
I20260620 12:22:14.137497 15165 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: Bootstrap starting.
I20260620 12:22:14.138212 15164 tablet_bootstrap.cc:654] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:14.139474 15164 log.cc:824] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:14.140095 15165 tablet_bootstrap.cc:654] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:14.140895 15165 log.cc:824] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:14.141615 15164 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: No bootstrap required, opened a new log
I20260620 12:22:14.141812 15164 ts_tablet_manager.cc:1397] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 12:22:14.142828 15165 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: No bootstrap required, opened a new log
I20260620 12:22:14.142997 15165 ts_tablet_manager.cc:1397] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:22:14.145689 15164 raft_consensus.cc:348] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.145962 15164 raft_consensus.cc:374] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:14.146011 15164 raft_consensus.cc:729] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd733c5e4274df287305058cb01f75e, State: Initialized, Role: FOLLOWER
I20260620 12:22:14.146461 15164 consensus_queue.cc:260] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.146621 15165 raft_consensus.cc:348] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.146875 15165 raft_consensus.cc:374] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:14.146930 15165 raft_consensus.cc:729] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa25929be13c48769952c5725dfb5c9f, State: Initialized, Role: FOLLOWER
I20260620 12:22:14.147158 15151 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
I20260620 12:22:14.147313 15165 consensus_queue.cc:260] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.147452 15164 ts_tablet_manager.cc:1428] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260620 12:22:14.147917 15019 heartbeater.cc:503] Master 127.0.98.62:37673 was elected leader, sending a full tablet report...
I20260620 12:22:14.148169 15165 ts_tablet_manager.cc:1428] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 12:22:14.190584 14882 tablet.cc:2369] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:22:14.202898 15169 raft_consensus.cc:482] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:14.203089 15169 raft_consensus.cc:504] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.204231 15169 leader_election.cc:288] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773), ea44ffd0be8f403bbf80850d5e69e175 (127.0.98.1:37991)
I20260620 12:22:14.208194 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.204475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37991 (local address 127.0.98.2:58499)
0620 12:22:14.204675 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.204693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:14.204781 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:14.205096 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:14.205100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:14.205121 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:14.205253 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.205262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.206104 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.206110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.206920 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.206929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.207766 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.207770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.207889 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.207916 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:22:14.207991 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:22:14.208068 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 12:22:14.208263 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.204619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58499 (local address 127.0.98.1:37991)
0620 12:22:14.204748 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:14.204753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:14.204772 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:22:14.204888 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:14.204891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:14.204960 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:14.205082 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:14.205371 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.205988 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.207068 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.207636 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.207730 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.207887 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:14.208023 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 12:22:14.208103 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 12:22:14.208438 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.204475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37773 (local address 127.0.98.2:35505)
0620 12:22:14.204925 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.204939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:14.205074 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:14.205339 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:14.205342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:14.205360 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:14.205473 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.205479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.206281 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.206284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.207192 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:14.207204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.207937 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.207949 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:14.208081 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.208118 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 12:22:14.208204 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 12:22:14.208273 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":43,"thread_start_us":123,"threads_started":1}
I20260620 12:22:14.208460 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.204590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35505 (local address 127.0.98.3:37773)
0620 12:22:14.204772 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:14.204777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:14.205052 (+ 275us) server_negotiation.cc:407] Connection header received
0620 12:22:14.205155 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:14.205158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:14.205211 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:14.205293 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:14.205580 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.206084 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.207330 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.207788 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.207898 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.207959 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:14.208239 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 12:22:14.208326 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 12:22:14.208869 14835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09768033336e4c8bbe11197d49b588f3" candidate_uuid: "fa25929be13c48769952c5725dfb5c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea44ffd0be8f403bbf80850d5e69e175" is_pre_election: true
I20260620 12:22:14.209124 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09768033336e4c8bbe11197d49b588f3" candidate_uuid: "fa25929be13c48769952c5725dfb5c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd733c5e4274df287305058cb01f75e" is_pre_election: true
I20260620 12:22:14.209223 14835 raft_consensus.cc:2454] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa25929be13c48769952c5725dfb5c9f in term 0.
I20260620 12:22:14.209404 15105 raft_consensus.cc:2454] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa25929be13c48769952c5725dfb5c9f in term 0.
I20260620 12:22:14.209739 14908 leader_election.cc:302] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea44ffd0be8f403bbf80850d5e69e175, fa25929be13c48769952c5725dfb5c9f; no voters:
I20260620 12:22:14.210055 15169 raft_consensus.cc:2781] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:22:14.210129 15169 raft_consensus.cc:482] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:22:14.210158 15169 raft_consensus.cc:3037] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:14.211206 15169 raft_consensus.cc:504] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.211603 15169 leader_election.cc:288] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [CANDIDATE]: Term 1 election: Requested vote from peers 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773), ea44ffd0be8f403bbf80850d5e69e175 (127.0.98.1:37991)
I20260620 12:22:14.211992 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09768033336e4c8bbe11197d49b588f3" candidate_uuid: "fa25929be13c48769952c5725dfb5c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd733c5e4274df287305058cb01f75e"
I20260620 12:22:14.212092 15105 raft_consensus.cc:3037] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:14.212146 14835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09768033336e4c8bbe11197d49b588f3" candidate_uuid: "fa25929be13c48769952c5725dfb5c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea44ffd0be8f403bbf80850d5e69e175"
I20260620 12:22:14.212289 14835 raft_consensus.cc:3037] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:14.213236 15105 raft_consensus.cc:2454] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa25929be13c48769952c5725dfb5c9f in term 1.
I20260620 12:22:14.213505 14909 leader_election.cc:302] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fd733c5e4274df287305058cb01f75e, fa25929be13c48769952c5725dfb5c9f; no voters:
I20260620 12:22:14.213511 14835 raft_consensus.cc:2454] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa25929be13c48769952c5725dfb5c9f in term 1.
I20260620 12:22:14.213693 15169 raft_consensus.cc:2781] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:14.213932 15169 raft_consensus.cc:686] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 1 LEADER]: Becoming Leader. State: Replica: fa25929be13c48769952c5725dfb5c9f, State: Running, Role: LEADER
I20260620 12:22:14.214296 15169 consensus_queue.cc:237] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.217659 14692 catalog_manager.cc:5495] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f reported cstate change: term changed from 0 to 1, leader changed from <none> to fa25929be13c48769952c5725dfb5c9f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fa25929be13c48769952c5725dfb5c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } health_report { overall_health: UNKNOWN } } }
I20260620 12:22:14.249996 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.245985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.0.98.2:44877)
0620 12:22:14.246150 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:14.246155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:14.246172 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:22:14.246430 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:14.246433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:14.246492 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:14.246622 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:22:14.246904 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.247458 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.248367 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:14.248617 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:14.248737 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:14.249100 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:22:14.249524 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:22:14.249713 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:14.249767 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 12:22:14.249821 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:22:14.255663 15105 raft_consensus.cc:1261] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 1 FOLLOWER]: Refusing update from remote peer fa25929be13c48769952c5725dfb5c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:14.255671 14835 raft_consensus.cc:1261] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Refusing update from remote peer fa25929be13c48769952c5725dfb5c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:14.256356 15171 consensus_queue.cc:1035] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:14.256543 15169 consensus_queue.cc:1035] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:14.262722 15174 mvcc.cc:205] Tried to move back new op lower bound from 7298900517904842752 to 7298900517744676864. Current Snapshot: MvccSnapshot[applied={T|T < 7298900517904842752}]
I20260620 12:22:14.270279 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:14.270439 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:14.272624 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Sending DeleteTablet for 3 replicas of tablet 09768033336e4c8bbe11197d49b588f3
I20260620 12:22:14.273245 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet 09768033336e4c8bbe11197d49b588f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60222
I20260620 12:22:14.273340 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet 09768033336e4c8bbe11197d49b588f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48374
I20260620 12:22:14.273330 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 09768033336e4c8bbe11197d49b588f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40310
I20260620 12:22:14.273664 15184 tablet_replica.cc:330] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: stopping tablet replica
I20260620 12:22:14.273793 15184 raft_consensus.cc:2229] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:14.273886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 14731
I20260620 12:22:14.273931 15184 raft_consensus.cc:2258] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:14.274153 15185 tablet_replica.cc:330] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: stopping tablet replica
I20260620 12:22:14.274545 15185 raft_consensus.cc:2229] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:22:14.274869 15185 raft_consensus.cc:2258] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:14.276311 15185 ts_tablet_manager.cc:1905] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:14.277246 15186 tablet_replica.cc:330] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: stopping tablet replica
I20260620 12:22:14.277593 15186 raft_consensus.cc:2229] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:14.277951 15186 raft_consensus.cc:2258] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:14.279042 15185 ts_tablet_manager.cc:1918] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:14.279146 15185 log.cc:1196] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/09768033336e4c8bbe11197d49b588f3
I20260620 12:22:14.279490 15185 ts_tablet_manager.cc:1939] T 09768033336e4c8bbe11197d49b588f3 P fa25929be13c48769952c5725dfb5c9f: Deleting consensus metadata
I20260620 12:22:14.279640 15186 ts_tablet_manager.cc:1905] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:14.280552 14678 catalog_manager.cc:4841] TS fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877): tablet 09768033336e4c8bbe11197d49b588f3 (table test-table [id=bbcff8978fad4d08bebceb563f99566a]) successfully deleted
W20260620 12:22:14.281984 14679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37991 (error 108)
I20260620 12:22:14.282790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 14885
I20260620 12:22:14.282929 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.282626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37991 (local address 127.0.0.1:48390)
0620 12:22:14.282781 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.282856 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 12:22:14.283891 15186 ts_tablet_manager.cc:1918] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 12:22:14.283397 14679 catalog_manager.cc:4584] TS ea44ffd0be8f403bbf80850d5e69e175 (127.0.98.1:37991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09768033336e4c8bbe11197d49b588f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37991: connect: Connection refused (error 111)
I20260620 12:22:14.284014 15186 log.cc:1196] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/09768033336e4c8bbe11197d49b588f3
I20260620 12:22:14.284322 15186 ts_tablet_manager.cc:1939] T 09768033336e4c8bbe11197d49b588f3 P 8fd733c5e4274df287305058cb01f75e: Deleting consensus metadata
I20260620 12:22:14.285298 14680 catalog_manager.cc:4841] TS 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773): tablet 09768033336e4c8bbe11197d49b588f3 (table test-table [id=bbcff8978fad4d08bebceb563f99566a]) successfully deleted
I20260620 12:22:14.288931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 15022
I20260620 12:22:14.294469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 14660
I20260620 12:22:14.300462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37673
--webserver_interface=127.0.98.62
--webserver_port=41383
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:14.405776 15188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:14.406020 15188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:14.406090 15188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:14.409986 15188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:14.410039 15188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:14.410058 15188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:14.410075 15188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:14.410090 15188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:14.415244 15188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:14.416290 15188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:14.417325 15188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:14.425705 15196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:14.425784 15188 server_base.cc:1034] running on GCE node
W20260620 12:22:14.425802 15193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:14.425796 15194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:14.426519 15188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:14.427562 15188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:14.428756 15188 hybrid_clock.cc:648] HybridClock initialized: now 1781958134428742 us; error 32 us; skew 500 ppm
Jun 20 12:22:14 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:14.431370 15188 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:14.432240 15188 webserver.cc:460] Webserver started at http://127.0.98.62:41383/ using document root <none> and password file <none>
I20260620 12:22:14.432789 15188 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:14.432878 15188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:14.436635 15188 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:14.438727 15203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:14.439863 15188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:22:14.439994 15188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "9dec88912b474ba98c9501fb6b82345b"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "ce3fbd9747092fcc754630442fac6978"
server_key_iv: "148f1b8ea2c76ec2cf59e20b6d11c4de"
server_key_version: "encryptionkey@0"
I20260620 12:22:14.440513 15188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:14.451644 15188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:14.454514 15188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:14.454713 15188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:14.461711 15188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37673
I20260620 12:22:14.461982 15255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37673 every 8 connection(s)
I20260620 12:22:14.462766 15188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:14.466471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 15188
I20260620 12:22:14.466936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37991
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37673
--webserver_port=36091
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:14.467068 15256 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:22:14.469342 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Bootstrap starting.
I20260620 12:22:14.486156 15256 log.cc:824] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:14.496743 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:22:14.497170 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b: Bootstrap complete.
I20260620 12:22:14.501155 15256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:14.501528 15256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec88912b474ba98c9501fb6b82345b, State: Initialized, Role: FOLLOWER
I20260620 12:22:14.502008 15256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:14.502115 15256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:22:14.502202 15256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:22:14.502315 15256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:22:14.504279 15256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:14.504590 15256 leader_election.cc:302] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dec88912b474ba98c9501fb6b82345b; no voters:
I20260620 12:22:14.504878 15256 leader_election.cc:288] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 12:22:14.504992 15260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 12:22:14.505242 15260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [term 3 LEADER]: Becoming Leader. State: Replica: 9dec88912b474ba98c9501fb6b82345b, State: Running, Role: LEADER
I20260620 12:22:14.505537 15260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } }
I20260620 12:22:14.506040 15256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:14.506448 15260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dec88912b474ba98c9501fb6b82345b. Latest consensus state: current_term: 3 leader_uuid: "9dec88912b474ba98c9501fb6b82345b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } } }
I20260620 12:22:14.506563 15260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:14.506829 15261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9dec88912b474ba98c9501fb6b82345b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dec88912b474ba98c9501fb6b82345b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37673 } } }
I20260620 12:22:14.506929 15261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dec88912b474ba98c9501fb6b82345b [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:14.508411 15264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:14.513612 15264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:14.514256 15264 catalog_manager.cc:1259] Loaded cluster ID: 3fd689694c6f4219a7841cb3f0a20cbe
I20260620 12:22:14.514314 15264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 12:22:14.515298 15264 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7801d5e9ad google::LogMessage::Fail() at ??:0
@ 0x7f7801d6188a google::LogMessage::SendToLog() at ??:0
@ 0x7f7801d5e4de google::LogMessage::Flush() at ??:0
@ 0x7f7801d5fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f780c69dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f780c69ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f780c69b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f780c6df640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7803fb03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7803fb0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7803fb23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7803fa1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f78055e56db start_thread at ??:0
@ 0x7f780085c61f clone at ??:0
W20260620 12:22:14.588157 15258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:14.588373 15258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:14.588415 15258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:14.592542 15258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:14.592617 15258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:14.592736 15258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:22:14.598011 15258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36091
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:14.599022 15258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:14.600023 15258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:14.609512 15297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:14.609512 15298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:14.609915 15258 server_base.cc:1034] running on GCE node
W20260620 12:22:14.610004 15300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:14.610386 15258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:14.611061 15258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:14.612257 15258 hybrid_clock.cc:648] HybridClock initialized: now 1781958134612241 us; error 38 us; skew 500 ppm
Jun 20 12:22:14 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:14.615051 15258 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:22:14.615995 15258 webserver.cc:460] Webserver started at http://127.0.98.1:36091/ using document root <none> and password file <none>
I20260620 12:22:14.616545 15258 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:14.616636 15258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:14.620405 15258 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:14.622520 15307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:14.623771 15258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:22:14.623858 15258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "ea44ffd0be8f403bbf80850d5e69e175"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "dcd2195223eb4ea176bd1d8ca848bbf3"
server_key_iv: "298297435c657e3ab66f9a9678d5b3d8"
server_key_version: "encryptionkey@0"
I20260620 12:22:14.624392 15258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:14.658121 15258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:14.661353 15258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:14.661557 15258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:14.662150 15258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:14.663246 15314 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:22:14.668689 15258 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:22:14.668771 15258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 12:22:14.668853 15258 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:22:14.670529 15258 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:22:14.670596 15258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 12:22:14.670908 15314 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Bootstrap starting.
I20260620 12:22:14.681070 15258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37991
I20260620 12:22:14.681133 15421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37991 every 8 connection(s)
I20260620 12:22:14.681995 15258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:22:14.683830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 15258
I20260620 12:22:14.684290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44877
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37673
--webserver_port=42161
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:14.692586 15314 log.cc:824] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:14.699038 15314 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:22:14.699389 15314 tablet_bootstrap.cc:492] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Bootstrap complete.
I20260620 12:22:14.700521 15314 ts_tablet_manager.cc:1397] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.004s
I20260620 12:22:14.702816 15314 raft_consensus.cc:348] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.703230 15314 raft_consensus.cc:729] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea44ffd0be8f403bbf80850d5e69e175, State: Initialized, Role: FOLLOWER
I20260620 12:22:14.703778 15314 consensus_queue.cc:260] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:14.704747 15314 ts_tablet_manager.cc:1428] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260620 12:22:14.764590 15424 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:22:14.683933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:43933)
0620 12:22:14.685126 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.685194 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:14.686245 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:14.763849 (+ 77604us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":144,"threads_started":1}
W20260620 12:22:14.765123 15422 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37673 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37673: Client connection negotiation failed: client connection to 127.0.98.62:37673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:22:14.765869 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.765633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:59087)
0620 12:22:14.765758 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.765814 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 12:22:14.766479 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.766284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:40615)
0620 12:22:14.766387 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.766436 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 12:22:14.766636 15422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:22:14.818220 15425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:14.818508 15425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:14.818583 15425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:14.822618 15425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:14.822692 15425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:14.822786 15425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:22:14.827935 15425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42161
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:14.828953 15425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:14.829974 15425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:14.839543 15432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:14.839599 15425 server_base.cc:1034] running on GCE node
W20260620 12:22:14.839540 15435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:14.839540 15433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:14.840166 15425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:14.840754 15425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:14.841936 15425 hybrid_clock.cc:648] HybridClock initialized: now 1781958134841922 us; error 25 us; skew 500 ppm
Jun 20 12:22:14 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:14.844688 15425 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:22:14.845552 15425 webserver.cc:460] Webserver started at http://127.0.98.2:42161/ using document root <none> and password file <none>
I20260620 12:22:14.846100 15425 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:14.846201 15425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:14.850133 15425 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:22:14.852109 15442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:14.853336 15425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:14.853461 15425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "fa25929be13c48769952c5725dfb5c9f"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "1053c75a19634d42c605b269d4649c06"
server_key_iv: "a447ed7106ad99575344f86cf880d697"
server_key_version: "encryptionkey@0"
I20260620 12:22:14.853984 15425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:14.868333 15425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:14.871294 15425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:14.871493 15425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:14.872061 15425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:14.872857 15425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:14.872931 15425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:14.872997 15425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:14.873046 15425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:14.881520 15425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44877
I20260620 12:22:14.881598 15555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44877 every 8 connection(s)
I20260620 12:22:14.882550 15425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:22:14.883469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 15425
I20260620 12:22:14.883934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37773
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37673
--webserver_port=36409
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:14.886545 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.884801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:45281)
0620 12:22:14.885289 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.885872 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
W20260620 12:22:14.887053 15556 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37673 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37673: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
I20260620 12:22:14.887799 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.887476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:46357)
0620 12:22:14.887644 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.887714 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 12:22:14.888470 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:14.888201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:52865)
0620 12:22:14.888334 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:14.888394 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 12:22:14.888645 15556 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:22:14.991664 15558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:14.991904 15558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:14.991945 15558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:14.995877 15558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:14.995929 15558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:14.995985 15558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:22:15.001016 15558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36409
--tserver_master_addrs=127.0.98.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:15.002014 15558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:15.002936 15558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:15.012272 15565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:15.012327 15564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:15.012423 15558 server_base.cc:1034] running on GCE node
W20260620 12:22:15.012346 15567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:15.012869 15558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:15.013484 15558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:15.014672 15558 hybrid_clock.cc:648] HybridClock initialized: now 1781958135014655 us; error 28 us; skew 500 ppm
Jun 20 12:22:15 dist-test-slave-9b3m krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:15.017324 15558 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:22:15.018136 15558 webserver.cc:460] Webserver started at http://127.0.98.3:36409/ using document root <none> and password file <none>
I20260620 12:22:15.018618 15558 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:15.018674 15558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:15.022222 15558 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:15.024123 15574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:15.025312 15558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:15.025406 15558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "8fd733c5e4274df287305058cb01f75e"
format_stamp: "Formatted at 2026-06-20 12:22:12 on dist-test-slave-9b3m"
server_key: "5d064318077892fdfb927494c2990e80"
server_key_iv: "4e3e551a53b8c6cca98b4ea31604becb"
server_key_version: "encryptionkey@0"
I20260620 12:22:15.025892 15558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:15.041258 15558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:15.044231 15558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:15.044368 15558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:15.044843 15558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:15.045562 15558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:15.045605 15558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:15.045643 15558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:15.045663 15558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:15.054242 15558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37773
I20260620 12:22:15.054281 15687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37773 every 8 connection(s)
I20260620 12:22:15.055166 15558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:22:15.059082 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:15.057380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:58299)
0620 12:22:15.057820 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:15.058403 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
W20260620 12:22:15.059500 15688 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37673 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37673: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
I20260620 12:22:15.060025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 15558
I20260620 12:22:15.060138 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:15.059890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:53907)
0620 12:22:15.059998 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:15.060052 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 12:22:15.060861 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:15.060590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:52789)
0620 12:22:15.060707 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:15.060796 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 12:22:15.061134 15688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:22:15.767598 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:15.767157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:58019)
0620 12:22:15.767431 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:15.767498 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260620 12:22:15.889667 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:15.889272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:54429)
0620 12:22:15.889507 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:15.889560 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260620 12:22:16.062091 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:16.061630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:47597)
0620 12:22:16.061902 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:16.061965 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260620 12:22:16.302255 15695 raft_consensus.cc:482] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:16.302439 15695 raft_consensus.cc:504] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:16.303514 15695 leader_election.cc:288] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877), 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773)
Jun 20 12:22:16 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 12:22:16 dist-test-slave-9b3m krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 12:22:16.317364 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:16.303688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44877 (local address 127.0.98.1:40233)
0620 12:22:16.303985 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:16.303999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:16.304108 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:16.305730 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:16.305736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:16.306399 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:16.306852 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:16.306866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:16.308418 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:16.308421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:16.309324 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:16.309334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:16.310052 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:16.310055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:16.310190 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:16.310932 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:16.310944 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:16.312435 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:16.314720 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:16.314725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:16.314738 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:16.315063 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:16.315457 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:16.315459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:16.315461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:16.315632 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:16.316319 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:16.316325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:16.316556 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 12:22:16.316749 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 12:22:16.317119 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":98,"thread_start_us":90,"threads_started":1}
I20260620 12:22:16.317399 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:16.304000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54181 (local address 127.0.98.3:37773)
0620 12:22:16.304358 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:16.304375 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:16.304407 (+ 32us) server_negotiation.cc:407] Connection header received
0620 12:22:16.305002 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:16.305006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:16.305383 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:16.305681 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:16.307058 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:16.308241 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:16.309478 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:16.309905 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:16.310066 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:16.313101 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:16.313110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:16.313168 (+ 58us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:16.313200 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:16.315157 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:16.315618 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:16.315623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:16.315628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:16.315764 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:16.316028 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:16.316030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:16.316031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:16.316473 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:16.316597 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:16.316918 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 12:22:16.317233 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":93,"threads_started":1}
I20260620 12:22:16.317409 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:16.303848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40233 (local address 127.0.98.2:44877)
0620 12:22:16.304166 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:16.304196 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:16.304246 (+ 50us) server_negotiation.cc:407] Connection header received
0620 12:22:16.304939 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:16.304946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:16.305384 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:16.305680 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:16.307050 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:16.308261 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:16.309450 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:16.309909 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:16.310101 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:16.312689 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:16.312696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:16.312711 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:16.312742 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:16.314602 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:16.315187 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:16.315191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:16.315196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:16.315362 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:16.315728 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:16.315730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:16.315732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:16.316207 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:16.316305 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:16.316893 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 12:22:16.317225 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":53,"threads_started":1}
I20260620 12:22:16.317399 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:16.303783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37773 (local address 127.0.98.1:54181)
0620 12:22:16.304171 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:16.304187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:16.304290 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:16.305730 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:16.305736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:16.306399 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:16.306853 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:16.306868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:16.308381 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:16.308384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:16.309330 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:16.309339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:16.310025 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:16.310028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:16.310196 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:16.310877 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:16.310893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:16.312884 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:16.315258 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:16.315264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:16.315268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:16.315521 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:16.315853 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:16.315856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:16.315858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:16.315942 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:16.316578 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:16.316581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:16.316669 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 12:22:16.316785 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 12:22:16.317213 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":302,"spinlock_wait_cycles":10368,"thread_start_us":52,"threads_started":1}
W20260620 12:22:16.320379 15311 leader_election.cc:341] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773): Not found: Tablet not found: 09768033336e4c8bbe11197d49b588f3
W20260620 12:22:16.320521 15309 leader_election.cc:341] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877): Not found: Tablet not found: 09768033336e4c8bbe11197d49b588f3
I20260620 12:22:16.320627 15309 leader_election.cc:302] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea44ffd0be8f403bbf80850d5e69e175; no voters: 8fd733c5e4274df287305058cb01f75e, fa25929be13c48769952c5725dfb5c9f
I20260620 12:22:16.320890 15695 raft_consensus.cc:2726] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:22:16.768666 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:16.768387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:36347)
0620 12:22:16.768528 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:16.768583 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 12:22:16.890772 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:16.890374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:55267)
0620 12:22:16.890635 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:16.890689 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260620 12:22:17.063158 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:17.062807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:34059)
0620 12:22:17.063024 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:17.063074 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260620 12:22:17.769836 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:17.769377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:37593)
0620 12:22:17.769678 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:17.769728 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":114,"threads_started":1}
I20260620 12:22:17.892153 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:17.891537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:46673)
0620 12:22:17.891921 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:17.892019 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":58,"threads_started":1}
I20260620 12:22:18.064231 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:18.063831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:52425)
0620 12:22:18.064079 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:18.064132 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
I20260620 12:22:18.134622 15705 raft_consensus.cc:482] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:18.134748 15705 raft_consensus.cc:504] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:18.135241 15705 leader_election.cc:288] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877), 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773)
W20260620 12:22:18.136047 15309 leader_election.cc:341] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877): Not found: Tablet not found: 09768033336e4c8bbe11197d49b588f3
W20260620 12:22:18.136132 15311 leader_election.cc:341] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773): Not found: Tablet not found: 09768033336e4c8bbe11197d49b588f3
I20260620 12:22:18.136247 15311 leader_election.cc:302] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea44ffd0be8f403bbf80850d5e69e175; no voters: 8fd733c5e4274df287305058cb01f75e, fa25929be13c48769952c5725dfb5c9f
I20260620 12:22:18.136381 15705 raft_consensus.cc:2726] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:22:18.770972 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:18.770561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:33869)
0620 12:22:18.770798 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:18.770854 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260620 12:22:18.893321 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:18.892868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:59815)
0620 12:22:18.893175 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:18.893232 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":73,"threads_started":1}
I20260620 12:22:19.065400 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:19.064974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:34393)
0620 12:22:19.065249 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:19.065299 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":129,"threads_started":1}
I20260620 12:22:19.772259 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:19.771826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:53889)
0620 12:22:19.772118 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:19.772169 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
I20260620 12:22:19.894500 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:19.894068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:45925)
0620 12:22:19.894350 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:19.894405 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260620 12:22:20.066470 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:20.066057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:37857)
0620 12:22:20.066316 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:20.066368 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260620 12:22:20.247320 15712 raft_consensus.cc:482] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:20.247439 15712 raft_consensus.cc:504] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:20.247901 15712 leader_election.cc:288] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877), 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773)
W20260620 12:22:20.248642 15309 leader_election.cc:341] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877): Not found: Tablet not found: 09768033336e4c8bbe11197d49b588f3
W20260620 12:22:20.248759 15311 leader_election.cc:341] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773): Not found: Tablet not found: 09768033336e4c8bbe11197d49b588f3
I20260620 12:22:20.248835 15311 leader_election.cc:302] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea44ffd0be8f403bbf80850d5e69e175; no voters: 8fd733c5e4274df287305058cb01f75e, fa25929be13c48769952c5725dfb5c9f
I20260620 12:22:20.248967 15712 raft_consensus.cc:2726] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:22:20.773341 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:20.772914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:34621)
0620 12:22:20.773186 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:20.773250 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":70,"threads_started":1}
I20260620 12:22:20.895663 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:20.895235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:42829)
0620 12:22:20.895512 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:20.895570 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260620 12:22:21.067649 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:21.067226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:32899)
0620 12:22:21.067492 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:21.067550 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260620 12:22:21.774477 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:21.774096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:43979)
0620 12:22:21.774330 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:21.774383 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260620 12:22:21.896904 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:21.896431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:45171)
0620 12:22:21.896765 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:21.896818 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260620 12:22:22.068758 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:22.068339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:57409)
0620 12:22:22.068617 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:22.068670 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":68,"threads_started":1}
I20260620 12:22:22.775591 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:22.775210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:57889)
0620 12:22:22.775440 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:22.775491 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260620 12:22:22.898245 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:22.897702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:42029)
0620 12:22:22.898026 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:22.898102 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260620 12:22:23.069872 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:23.069461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:47995)
0620 12:22:23.069738 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:23.069788 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":127,"threads_started":1}
I20260620 12:22:23.237425 15722 raft_consensus.cc:482] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:23.237560 15722 raft_consensus.cc:504] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa25929be13c48769952c5725dfb5c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44877 } } peers { permanent_uuid: "8fd733c5e4274df287305058cb01f75e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37773 } } peers { permanent_uuid: "ea44ffd0be8f403bbf80850d5e69e175" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37991 } }
I20260620 12:22:23.237959 15722 leader_election.cc:288] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877), 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773)
W20260620 12:22:23.238691 15311 leader_election.cc:341] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fd733c5e4274df287305058cb01f75e (127.0.98.3:37773): Not found: Tablet not found: 09768033336e4c8bbe11197d49b588f3
W20260620 12:22:23.238788 15309 leader_election.cc:341] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa25929be13c48769952c5725dfb5c9f (127.0.98.2:44877): Not found: Tablet not found: 09768033336e4c8bbe11197d49b588f3
I20260620 12:22:23.238899 15309 leader_election.cc:302] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea44ffd0be8f403bbf80850d5e69e175; no voters: 8fd733c5e4274df287305058cb01f75e, fa25929be13c48769952c5725dfb5c9f
I20260620 12:22:23.239048 15722 raft_consensus.cc:2726] T 09768033336e4c8bbe11197d49b588f3 P ea44ffd0be8f403bbf80850d5e69e175 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:22:23.776752 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:23.776313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:45899)
0620 12:22:23.776601 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:23.776657 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260620 12:22:23.899384 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:23.898993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:57279)
0620 12:22:23.899243 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:23.899294 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260620 12:22:24.071048 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:24.070654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:58343)
0620 12:22:24.070905 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:24.070959 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260620 12:22:24.778112 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:24.777707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:60185)
0620 12:22:24.777965 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:24.778020 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":72,"threads_started":1}
I20260620 12:22:24.900535 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:24.900126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:56127)
0620 12:22:24.900395 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:24.900447 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
I20260620 12:22:25.072280 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:25.071899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:40907)
0620 12:22:25.072142 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:25.072192 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
I20260620 12:22:25.779340 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:25.778934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:46303)
0620 12:22:25.779192 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:25.779246 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260620 12:22:25.901743 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:25.901340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:51827)
0620 12:22:25.901590 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:25.901645 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260620 12:22:26.073442 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:26.072942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:39463)
0620 12:22:26.073289 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:26.073349 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":69,"threads_started":1}
I20260620 12:22:26.780476 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:26.780074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:39877)
0620 12:22:26.780328 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:26.780383 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260620 12:22:26.902930 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:26.902537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:55137)
0620 12:22:26.902785 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:26.902838 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
I20260620 12:22:27.074590 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:27.074207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:59051)
0620 12:22:27.074447 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:27.074501 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260620 12:22:27.781572 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:27.781187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:48621)
0620 12:22:27.781426 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:27.781478 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
I20260620 12:22:27.904393 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:27.903813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:55745)
0620 12:22:27.904163 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:27.904238 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20260620 12:22:28.075696 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:28.075301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:46971)
0620 12:22:28.075547 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:28.075601 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260620 12:22:28.782835 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:28.782425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:40075)
0620 12:22:28.782682 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:28.782737 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260620 12:22:28.905663 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:28.905259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:36151)
0620 12:22:28.905509 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:28.905567 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260620 12:22:29.076960 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:29.076524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.3:51725)
0620 12:22:29.076818 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:29.076869 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":107,"threads_started":1}
I20260620 12:22:29.784214 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:29.783791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.1:55711)
0620 12:22:29.784065 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:29.784122 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":69,"threads_started":1}
I20260620 12:22:29.907020 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:29.906584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37673 (local address 127.0.98.2:55191)
0620 12:22:29.906832 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:29.906892 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260620 12:22:30.063405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 15258
I20260620 12:22:30.069357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 15425
I20260620 12:22:30.074529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 15558
2026-06-20T12:22:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19259 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:22:30 dist-test-slave-9b3m krb5kdc[15745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:22:30 dist-test-slave-9b3m krb5kdc[15745](info): set up 2 sockets
Jun 20 12:22:30 dist-test-slave-9b3m krb5kdc[15745](info): commencing operation
krb5kdc: starting...
W20260620 12:22:31.121465 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:22:31 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:22:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:22:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:22:31.244920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42469
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:31.352550 15758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:31.352877 15758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:31.352962 15758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:31.356987 15758 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:22:31.357095 15758 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:22:31.357128 15758 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:22:31.357160 15758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:31.357197 15758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:31.357259 15758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:31.357296 15758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:31.357332 15758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:31.362567 15758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:31.363633 15758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:31.364671 15758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:31.373677 15766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:31.373768 15764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:31.373677 15763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:31.373742 15758 server_base.cc:1034] running on GCE node
I20260620 12:22:31.374655 15758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:31.375722 15758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:31.376933 15758 hybrid_clock.cc:648] HybridClock initialized: now 1781958151376911 us; error 34 us; skew 500 ppm
Jun 20 12:22:31 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:31.379657 15758 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:31.380537 15758 webserver.cc:460] Webserver started at http://127.0.98.62:45893/ using document root <none> and password file <none>
I20260620 12:22:31.381027 15758 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:31.381143 15758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:31.381316 15758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:31.383210 15758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "88580d6ebedc419aacbf1bf833553ee9"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "4a2dc163cadec2d156158aaad4769e6d"
server_key_iv: "292bfbe05f39138dbae680f92d15c515"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.383708 15758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "88580d6ebedc419aacbf1bf833553ee9"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "4a2dc163cadec2d156158aaad4769e6d"
server_key_iv: "292bfbe05f39138dbae680f92d15c515"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.387218 15758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:22:31.389631 15773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:31.390885 15758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:31.391021 15758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "88580d6ebedc419aacbf1bf833553ee9"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "4a2dc163cadec2d156158aaad4769e6d"
server_key_iv: "292bfbe05f39138dbae680f92d15c515"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.391146 15758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:31.412582 15758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:31.415488 15758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:31.415679 15758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:31.422559 15758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42469
I20260620 12:22:31.422851 15825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42469 every 8 connection(s)
I20260620 12:22:31.423534 15758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:31.426357 15826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:31.431282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 15758
I20260620 12:22:31.431386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:22:31.431761 392 external_mini_cluster.cc:1428] Setting key 6007eb49e0f4e8fb7c3fa080fe5cb447
I20260620 12:22:31.431946 15826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Bootstrap starting.
I20260620 12:22:31.434706 15826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:31.435580 15826 log.cc:824] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:31.437772 15826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: No bootstrap required, opened a new log
Jun 20 12:22:31 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:31.441936 15826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:22:31.442216 15826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:31.442272 15826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88580d6ebedc419aacbf1bf833553ee9, State: Initialized, Role: FOLLOWER
I20260620 12:22:31.442775 15826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:22:31.442962 15826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:22:31.443037 15826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:22:31.443145 15826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:31.444479 15826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:22:31.444885 15826 leader_election.cc:302] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88580d6ebedc419aacbf1bf833553ee9; no voters:
I20260620 12:22:31.445282 15826 leader_election.cc:288] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:22:31.445436 15831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:31.445686 15831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 1 LEADER]: Becoming Leader. State: Replica: 88580d6ebedc419aacbf1bf833553ee9, State: Running, Role: LEADER
I20260620 12:22:31.446089 15831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:22:31.446583 15826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:31.447440 15831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88580d6ebedc419aacbf1bf833553ee9. Latest consensus state: current_term: 1 leader_uuid: "88580d6ebedc419aacbf1bf833553ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } } }
I20260620 12:22:31.447672 15831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:31.447927 15832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88580d6ebedc419aacbf1bf833553ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } } }
I20260620 12:22:31.448068 15832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:31.448140 15840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:31.448661 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:31.433215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.0.98.62:42469)
0620 12:22:31.433639 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:31.433657 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:31.433698 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:22:31.434511 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:31.434534 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:31.434968 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:31.435346 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:31.436067 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.437493 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.438529 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.439196 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.439366 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:31.441990 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:31.441997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:31.442013 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:31.442044 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:31.444458 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:31.445032 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:31.445038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:31.445063 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:31.445220 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:31.445605 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:31.445612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:31.445616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:31.446185 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:31.446411 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:31.447208 (+ 797us) server_negotiation.cc:299] Negotiation successful
0620 12:22:31.447552 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
I20260620 12:22:31.451886 15840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:31.458580 15840 catalog_manager.cc:1347] Generated new cluster ID: 1073eac9a25e4b778ca634de9af44951
I20260620 12:22:31.458760 15840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:22:31.466917 15840 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:22:31.467864 15840 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:22:31.481566 15840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Generated new TSK 0
I20260620 12:22:31.482272 15840 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:22:31.515978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42469
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:31.645144 15852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:31.645386 15852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:31.645428 15852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:31.649904 15852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:31.649966 15852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:31.650050 15852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:22:31.655390 15852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:31.656456 15852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:31.657517 15852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:31.667091 15857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:31.667104 15858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:31.667124 15860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:31.667418 15852 server_base.cc:1034] running on GCE node
I20260620 12:22:31.667963 15852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:31.668496 15852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:31.669677 15852 hybrid_clock.cc:648] HybridClock initialized: now 1781958151669660 us; error 31 us; skew 500 ppm
Jun 20 12:22:31 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:31.672463 15852 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:22:31.673389 15852 webserver.cc:460] Webserver started at http://127.0.98.1:35331/ using document root <none> and password file <none>
I20260620 12:22:31.673946 15852 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:31.674034 15852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:31.674228 15852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:31.676141 15852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "d5f59ea98b814bce824ad32365cf2737"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "2b7e7dba81d5028f4434b19d89221400"
server_key_iv: "45cb0da1c4346e5cbbafda11e4c2defd"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.676673 15852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d5f59ea98b814bce824ad32365cf2737"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "2b7e7dba81d5028f4434b19d89221400"
server_key_iv: "45cb0da1c4346e5cbbafda11e4c2defd"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.680624 15852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 12:22:31.683110 15867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:31.684288 15852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:31.684440 15852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "d5f59ea98b814bce824ad32365cf2737"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "2b7e7dba81d5028f4434b19d89221400"
server_key_iv: "45cb0da1c4346e5cbbafda11e4c2defd"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.684552 15852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:31.703172 15852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:31.705952 15852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:31.706151 15852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:31.706713 15852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:31.707515 15852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:31.707592 15852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:31.707661 15852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:31.707705 15852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:31.716657 15852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44259
I20260620 12:22:31.716678 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44259 every 8 connection(s)
I20260620 12:22:31.717541 15852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:22:31.723259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 15852
I20260620 12:22:31.723419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:22:31.723740 392 external_mini_cluster.cc:1428] Setting key 01545790abff28a56e1e9bb7a3083e2a
Jun 20 12:22:31 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:31.731572 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:31.719329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49269 (local address 127.0.98.62:42469)
0620 12:22:31.719470 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:31.719473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:31.720470 (+ 997us) server_negotiation.cc:407] Connection header received
0620 12:22:31.721434 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:31.721438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:31.721498 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:31.721604 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:31.722749 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.723310 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.724857 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.725181 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.725303 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:31.727683 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:31.727686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:31.727689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:31.727715 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:31.729228 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:31.729848 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:31.729851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:31.729852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:31.729926 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:31.730347 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:31.730350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:31.730351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:31.730550 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:31.730689 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:31.731347 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 12:22:31.731453 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:22:31.732724 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:31.719660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:49269)
0620 12:22:31.720309 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:31.720349 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:31.721228 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:31.721770 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:31.721777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:31.722147 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:31.722601 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:31.722614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.723484 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.723487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:31.724726 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:31.724736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.725331 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.725336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:31.725502 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:31.726089 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:31.726105 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:31.727520 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:31.729340 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:31.729348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:31.729365 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:31.729737 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:31.730018 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:31.730023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:31.730028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:31.730223 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:31.730713 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:31.730722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:31.731063 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 12:22:31.731283 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 12:22:31.731636 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:22:31.734036 15981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42469
I20260620 12:22:31.734269 15981 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:31.734771 15981 heartbeater.cc:511] Master 127.0.98.62:42469 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:22:31.736397 15790 ts_manager.cc:194] Registered new tserver with Master: d5f59ea98b814bce824ad32365cf2737 (127.0.98.1:44259)
I20260620 12:22:31.737584 15790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:22:31.752749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42469
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:31.860239 15986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:31.860492 15986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:31.860535 15986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:31.864749 15986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:31.864807 15986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:31.864894 15986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:22:31.870133 15986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:31.871178 15986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:31.872202 15986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:31.882103 15992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:31.882108 15991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:31.882110 15994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:31.882377 15986 server_base.cc:1034] running on GCE node
I20260620 12:22:31.882930 15986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:31.883556 15986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:31.884740 15986 hybrid_clock.cc:648] HybridClock initialized: now 1781958151884726 us; error 28 us; skew 500 ppm
Jun 20 12:22:31 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:31.887645 15986 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:22:31.888648 15986 webserver.cc:460] Webserver started at http://127.0.98.2:35917/ using document root <none> and password file <none>
I20260620 12:22:31.889302 15986 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:31.889369 15986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:31.889576 15986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:31.891556 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "01f32d93042c4468803a462695e906e6"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "01ed41f037db96cf5294063ad47a1c38"
server_key_iv: "e1f4c00b86d35c39922a0803cd0a242e"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.892084 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "01f32d93042c4468803a462695e906e6"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "01ed41f037db96cf5294063ad47a1c38"
server_key_iv: "e1f4c00b86d35c39922a0803cd0a242e"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.895706 15986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 12:22:31.898135 16001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:31.899358 15986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:31.899484 15986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "01f32d93042c4468803a462695e906e6"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "01ed41f037db96cf5294063ad47a1c38"
server_key_iv: "e1f4c00b86d35c39922a0803cd0a242e"
server_key_version: "encryptionkey@0"
I20260620 12:22:31.899592 15986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:31.917347 15986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:31.920482 15986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:31.920745 15986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:31.921355 15986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:31.922171 15986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:31.922242 15986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:31.922315 15986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:31.922360 15986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:31.931002 15986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35615
I20260620 12:22:31.931033 16114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35615 every 8 connection(s)
I20260620 12:22:31.931886 15986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:22:31.938862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 15986
I20260620 12:22:31.938998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:22:31.939357 392 external_mini_cluster.cc:1428] Setting key 2bc76bda1df1bce578be2c10fe503612
Jun 20 12:22:31 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:31.945627 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:31.933777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48515 (local address 127.0.98.62:42469)
0620 12:22:31.933932 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:31.933937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:31.934942 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 12:22:31.935800 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:31.935803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:31.935848 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:31.935922 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:31.937088 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.937634 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.938757 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.939137 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.939249 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:31.941916 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:31.941922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:31.941928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:31.941961 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:31.943425 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:31.944040 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:31.944043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:31.944044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:31.944115 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:31.944502 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:31.944504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:31.944505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:31.944681 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:31.944754 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:31.945418 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 12:22:31.945513 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:22:31.946946 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:31.934142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:48515)
0620 12:22:31.934794 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:31.934835 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:31.935587 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:31.936094 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:31.936100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:31.936452 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:31.936913 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:31.936926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.937749 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.937752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:31.938635 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:31.938644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:31.939296 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:31.939299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:31.939487 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:31.940124 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:31.940141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:31.941766 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:31.943540 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:31.943548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:31.943564 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:31.943938 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:31.944202 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:31.944208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:31.944213 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:31.944401 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:31.944803 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:31.944811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:31.945161 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 12:22:31.945385 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 12:22:31.945739 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":115,"threads_started":1}
I20260620 12:22:31.948256 16115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42469
I20260620 12:22:31.948546 16115 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:31.949088 16115 heartbeater.cc:511] Master 127.0.98.62:42469 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:22:31.950066 15790 ts_manager.cc:194] Registered new tserver with Master: 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615)
I20260620 12:22:31.950683 15790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:22:31.967669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42469
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:32.078471 16120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:32.078742 16120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:32.078815 16120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:32.082756 16120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:32.082832 16120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:32.082923 16120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:22:32.088101 16120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:32.089226 16120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:32.090209 16120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:32.099831 16125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:32.099831 16126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:32.099934 16128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:32.100013 16120 server_base.cc:1034] running on GCE node
I20260620 12:22:32.100535 16120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:32.101198 16120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:32.102391 16120 hybrid_clock.cc:648] HybridClock initialized: now 1781958152102371 us; error 28 us; skew 500 ppm
Jun 20 12:22:32 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:32.105340 16120 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:22:32.106185 16120 webserver.cc:460] Webserver started at http://127.0.98.3:42075/ using document root <none> and password file <none>
I20260620 12:22:32.106734 16120 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:32.106794 16120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:32.106999 16120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:22:32.108937 16120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "482b78f579214863bfa10951d3c3888e"
format_stamp: "Formatted at 2026-06-20 12:22:32 on dist-test-slave-9b3m"
server_key: "c8431ed9322785f71cc4fe3bac272b62"
server_key_iv: "32f5cacc51633304864f93df07274109"
server_key_version: "encryptionkey@0"
I20260620 12:22:32.109555 16120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "482b78f579214863bfa10951d3c3888e"
format_stamp: "Formatted at 2026-06-20 12:22:32 on dist-test-slave-9b3m"
server_key: "c8431ed9322785f71cc4fe3bac272b62"
server_key_iv: "32f5cacc51633304864f93df07274109"
server_key_version: "encryptionkey@0"
I20260620 12:22:32.113109 16120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:22:32.115424 16135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:32.116600 16120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:32.116740 16120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "482b78f579214863bfa10951d3c3888e"
format_stamp: "Formatted at 2026-06-20 12:22:32 on dist-test-slave-9b3m"
server_key: "c8431ed9322785f71cc4fe3bac272b62"
server_key_iv: "32f5cacc51633304864f93df07274109"
server_key_version: "encryptionkey@0"
I20260620 12:22:32.116858 16120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:32.125981 16120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:32.128683 16120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:32.128880 16120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:32.129487 16120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:32.130234 16120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:32.130312 16120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:32.130375 16120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:32.130427 16120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:32.139451 16120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41851
I20260620 12:22:32.139470 16248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41851 every 8 connection(s)
I20260620 12:22:32.140293 16120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:22:32.143499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 16120
I20260620 12:22:32.143656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:22:32.143942 392 external_mini_cluster.cc:1428] Setting key e26934f3180dafdd36eed411860d0148
Jun 20 12:22:32 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:32.153815 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.142107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37683 (local address 127.0.98.62:42469)
0620 12:22:32.142239 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:32.142249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:32.143002 (+ 753us) server_negotiation.cc:407] Connection header received
0620 12:22:32.144384 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:32.144387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:32.144465 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:32.144556 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:32.146085 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.146628 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.147919 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.148184 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.148281 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.150609 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:32.150612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:32.150614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:32.150638 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:32.152098 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:32.152507 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:32.152509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:32.152510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:32.152579 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:32.152875 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:32.152877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:32.152878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:32.153105 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:32.153180 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:32.153601 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 12:22:32.153694 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:22:32.154604 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.142429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:37683)
0620 12:22:32.142848 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:32.142888 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:32.144153 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:32.144766 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:32.144775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:32.145280 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:32.145939 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.145959 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.146731 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.146733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.147760 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.147769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.148336 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.148346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.148482 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.149016 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:32.149030 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:32.150467 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:32.152189 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:32.152192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:32.152203 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:32.152426 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:32.152652 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:32.152654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:32.152656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:32.152798 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:32.153188 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:32.153192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:32.153404 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 12:22:32.153555 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 12:22:32.153831 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260620 12:22:32.155637 16249 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42469
I20260620 12:22:32.155918 16249 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:32.156443 16249 heartbeater.cc:511] Master 127.0.98.62:42469 requested a full tablet report, sending...
I20260620 12:22:32.157612 15789 ts_manager.cc:194] Registered new tserver with Master: 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851)
I20260620 12:22:32.158200 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37683
I20260620 12:22:32.169409 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:22:32.177910 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.171299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.0.98.62:42469)
0620 12:22:32.171422 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:32.171425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:32.171550 (+ 125us) server_negotiation.cc:407] Connection header received
0620 12:22:32.171645 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:32.171647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:32.171682 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:32.171767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:32.172388 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.172890 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.173688 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.173908 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.173985 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.175342 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:32.175345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:32.175347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:32.175365 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:32.176521 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:32.176889 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:32.176891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:32.176892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:32.176961 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:32.177198 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:32.177200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:32.177201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:32.177400 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:32.177499 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:32.177686 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 12:22:32.177797 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:22:32.180789 15789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:22:32.183789 15789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:22:32.198608 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.191749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41851 (local address 127.0.0.1:51668)
0620 12:22:32.192503 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:32.192555 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:32.192761 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:32.193805 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:32.193839 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:32.193867 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:32.194050 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.194058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.195549 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.195552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.196979 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.196988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.198094 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.198098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.198205 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.198244 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 12:22:32.198364 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 12:22:32.198469 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20260620 12:22:32.198684 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.191950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.0.98.3:41851)
0620 12:22:32.192409 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:32.192421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:32.192639 (+ 218us) server_negotiation.cc:407] Connection header received
0620 12:22:32.192979 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:32.192985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:32.193617 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:32.193807 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:32.194187 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.195370 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.197139 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.197964 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.198078 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.198233 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:32.198406 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 12:22:32.198497 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":121,"threads_started":1}
I20260620 12:22:32.198869 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.192340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.0.98.2:35615)
0620 12:22:32.192745 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:32.192784 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:32.193397 (+ 613us) server_negotiation.cc:407] Connection header received
0620 12:22:32.193475 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:32.193482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:32.193684 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:32.193851 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:32.194279 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.195213 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.197329 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.198082 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.198174 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.198302 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:32.198635 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 12:22:32.198734 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":115,"threads_started":1}
I20260620 12:22:32.199347 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.192150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.0.1:60322)
0620 12:22:32.193195 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:32.193209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:32.193305 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:32.194040 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:32.194043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:32.194062 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:32.194152 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.194158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.195339 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.195346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.197197 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.197207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.198391 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.198394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.198482 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.198504 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:22:32.199120 (+ 616us) client_negotiation.cc:240] Negotiation successful
0620 12:22:32.199218 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":103,"threads_started":1}
I20260620 12:22:32.200398 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.194389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44259 (local address 127.0.0.1:53002)
0620 12:22:32.195691 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:32.195705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:32.195811 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:32.196475 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:32.196478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:32.196493 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:32.196562 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.196567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.197766 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.197772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.198816 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.198826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.200018 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.200021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.200110 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.200130 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 12:22:32.200215 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 12:22:32.200281 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"thread_start_us":406,"threads_started":1}
I20260620 12:22:32.200446 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.193148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.0.98.1:44259)
0620 12:22:32.194100 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:32.194117 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:32.195912 (+ 1795us) server_negotiation.cc:407] Connection header received
0620 12:22:32.195994 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:32.196000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:32.196178 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:32.196316 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:32.196676 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.197627 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.198928 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.199896 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.200018 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.200154 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:32.200236 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 12:22:32.200318 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":120,"threads_started":1}
I20260620 12:22:32.202217 16049 tablet_service.cc:1467] Processing CreateTablet for tablet 6727935cf80741bb8967d0b8b1c15231 (DEFAULT_TABLE table=test-table [id=564baf07695a4f6fa59d692be66f2078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:32.202231 16183 tablet_service.cc:1467] Processing CreateTablet for tablet 6727935cf80741bb8967d0b8b1c15231 (DEFAULT_TABLE table=test-table [id=564baf07695a4f6fa59d692be66f2078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:32.203331 16049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6727935cf80741bb8967d0b8b1c15231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:32.203334 16183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6727935cf80741bb8967d0b8b1c15231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:32.211256 16267 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Bootstrap starting.
I20260620 12:22:32.211645 16266 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Bootstrap starting.
I20260620 12:22:32.211663 15915 tablet_service.cc:1467] Processing CreateTablet for tablet 6727935cf80741bb8967d0b8b1c15231 (DEFAULT_TABLE table=test-table [id=564baf07695a4f6fa59d692be66f2078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:32.212692 15915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6727935cf80741bb8967d0b8b1c15231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:32.213924 16266 tablet_bootstrap.cc:654] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:32.214556 16267 tablet_bootstrap.cc:654] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:32.214653 16266 log.cc:824] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:32.215368 16267 log.cc:824] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:32.216609 16266 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: No bootstrap required, opened a new log
I20260620 12:22:32.216814 16266 ts_tablet_manager.cc:1397] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:22:32.217265 16267 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: No bootstrap required, opened a new log
I20260620 12:22:32.217440 16267 ts_tablet_manager.cc:1397] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:22:32.219898 16270 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Bootstrap starting.
I20260620 12:22:32.219981 16266 raft_consensus.cc:348] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.220288 16266 raft_consensus.cc:374] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:32.220366 16266 raft_consensus.cc:729] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f32d93042c4468803a462695e906e6, State: Initialized, Role: FOLLOWER
I20260620 12:22:32.220834 16266 consensus_queue.cc:260] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.221364 16267 raft_consensus.cc:348] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.221662 16267 raft_consensus.cc:374] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:32.221732 16267 raft_consensus.cc:729] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482b78f579214863bfa10951d3c3888e, State: Initialized, Role: FOLLOWER
I20260620 12:22:32.221700 16115 heartbeater.cc:503] Master 127.0.98.62:42469 was elected leader, sending a full tablet report...
I20260620 12:22:32.221874 16266 ts_tablet_manager.cc:1428] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:22:32.222146 16267 consensus_queue.cc:260] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.222855 16249 heartbeater.cc:503] Master 127.0.98.62:42469 was elected leader, sending a full tablet report...
I20260620 12:22:32.222927 16267 ts_tablet_manager.cc:1428] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:22:32.224077 16270 tablet_bootstrap.cc:654] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:32.225304 16270 log.cc:824] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:32.227108 16270 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: No bootstrap required, opened a new log
I20260620 12:22:32.227295 16270 ts_tablet_manager.cc:1397] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260620 12:22:32.229995 16270 raft_consensus.cc:348] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.230180 16270 raft_consensus.cc:374] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:32.230247 16270 raft_consensus.cc:729] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f59ea98b814bce824ad32365cf2737, State: Initialized, Role: FOLLOWER
I20260620 12:22:32.230602 16270 consensus_queue.cc:260] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.231284 15981 heartbeater.cc:503] Master 127.0.98.62:42469 was elected leader, sending a full tablet report...
I20260620 12:22:32.231526 16270 ts_tablet_manager.cc:1428] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260620 12:22:32.391394 16250 tablet.cc:2369] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:22:32.396404 16274 raft_consensus.cc:482] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:32.396642 16274 raft_consensus.cc:504] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.397852 16274 leader_election.cc:288] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851), 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615)
I20260620 12:22:32.403142 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.398279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59127 (local address 127.0.98.2:35615)
0620 12:22:32.398440 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:32.398445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:32.398575 (+ 130us) server_negotiation.cc:407] Connection header received
0620 12:22:32.398740 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:32.398743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:32.398803 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:32.398896 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:32.399353 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.400127 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.401888 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.402494 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.402603 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.402666 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:32.402934 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 12:22:32.403022 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:22:32.403142 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.398287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33169 (local address 127.0.98.3:41851)
0620 12:22:32.398455 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:32.398460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:32.398705 (+ 245us) server_negotiation.cc:407] Connection header received
0620 12:22:32.398787 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:32.398792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:32.398842 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:32.398915 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:32.399353 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.400127 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.401888 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.402494 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.402603 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.402685 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:32.402934 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 12:22:32.403012 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 12:22:32.403203 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.398119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.98.1:59127)
0620 12:22:32.398462 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:32.398481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:32.398618 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:32.398940 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:32.398944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:32.398987 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:32.399230 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.399240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.400246 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.400250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.401757 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.401768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.402627 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.402631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.402765 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.402818 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 12:22:32.402899 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 12:22:32.403023 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":263,"spinlock_wait_cycles":6784,"thread_start_us":100,"threads_started":1}
I20260620 12:22:32.403225 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.398119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41851 (local address 127.0.98.1:33169)
0620 12:22:32.398604 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:32.398616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:32.398704 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:32.398940 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:32.398943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:32.398986 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:32.399244 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.399249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.400246 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.400250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.401760 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:32.401768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.402627 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.402631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:32.402769 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.402820 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 12:22:32.402899 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:22:32.403022 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":201,"spinlock_wait_cycles":31488,"thread_start_us":129,"threads_started":1}
I20260620 12:22:32.403893 16203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6727935cf80741bb8967d0b8b1c15231" candidate_uuid: "d5f59ea98b814bce824ad32365cf2737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482b78f579214863bfa10951d3c3888e" is_pre_election: true
I20260620 12:22:32.404156 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6727935cf80741bb8967d0b8b1c15231" candidate_uuid: "d5f59ea98b814bce824ad32365cf2737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f32d93042c4468803a462695e906e6" is_pre_election: true
I20260620 12:22:32.404215 16203 raft_consensus.cc:2454] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f59ea98b814bce824ad32365cf2737 in term 0.
I20260620 12:22:32.404422 16069 raft_consensus.cc:2454] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f59ea98b814bce824ad32365cf2737 in term 0.
I20260620 12:22:32.404740 15868 leader_election.cc:302] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 482b78f579214863bfa10951d3c3888e, d5f59ea98b814bce824ad32365cf2737; no voters:
I20260620 12:22:32.405054 16274 raft_consensus.cc:2781] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:22:32.405143 16274 raft_consensus.cc:482] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:22:32.405195 16274 raft_consensus.cc:3037] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:32.406216 16274 raft_consensus.cc:504] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.406569 16274 leader_election.cc:288] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 1 election: Requested vote from peers 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851), 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615)
I20260620 12:22:32.406899 16203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6727935cf80741bb8967d0b8b1c15231" candidate_uuid: "d5f59ea98b814bce824ad32365cf2737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482b78f579214863bfa10951d3c3888e"
I20260620 12:22:32.406994 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6727935cf80741bb8967d0b8b1c15231" candidate_uuid: "d5f59ea98b814bce824ad32365cf2737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f32d93042c4468803a462695e906e6"
I20260620 12:22:32.406999 16203 raft_consensus.cc:3037] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:32.407130 16069 raft_consensus.cc:3037] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:32.408087 16203 raft_consensus.cc:2454] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f59ea98b814bce824ad32365cf2737 in term 1.
I20260620 12:22:32.408195 16069 raft_consensus.cc:2454] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f59ea98b814bce824ad32365cf2737 in term 1.
I20260620 12:22:32.408411 15868 leader_election.cc:302] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 482b78f579214863bfa10951d3c3888e, d5f59ea98b814bce824ad32365cf2737; no voters:
I20260620 12:22:32.408630 16274 raft_consensus.cc:2781] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:32.408811 16274 raft_consensus.cc:686] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 1 LEADER]: Becoming Leader. State: Replica: d5f59ea98b814bce824ad32365cf2737, State: Running, Role: LEADER
I20260620 12:22:32.409184 16274 consensus_queue.cc:237] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:32.412344 15789 catalog_manager.cc:5495] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5f59ea98b814bce824ad32365cf2737 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d5f59ea98b814bce824ad32365cf2737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } health_report { overall_health: UNKNOWN } } }
W20260620 12:22:32.433081 16116 tablet.cc:2369] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:22:32.433470 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.429744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.0.98.1:44259)
0620 12:22:32.429884 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:32.429887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:32.429901 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:22:32.429972 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:32.429974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:32.430030 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:32.430113 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:22:32.430362 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.430850 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.432035 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:32.432248 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:32.432323 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:32.432624 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:22:32.433017 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:22:32.433202 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:32.433247 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 12:22:32.433303 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:22:32.439702 16069 raft_consensus.cc:1261] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 1 FOLLOWER]: Refusing update from remote peer d5f59ea98b814bce824ad32365cf2737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:32.439702 16203 raft_consensus.cc:1261] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 1 FOLLOWER]: Refusing update from remote peer d5f59ea98b814bce824ad32365cf2737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:32.440363 16274 consensus_queue.cc:1035] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:32.440622 16277 consensus_queue.cc:1035] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:32.459007 15788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:32.459193 15788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:32.461544 15788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Sending DeleteTablet for 3 replicas of tablet 6727935cf80741bb8967d0b8b1c15231
I20260620 12:22:32.462404 16183 tablet_service.cc:1514] Processing DeleteTablet for tablet 6727935cf80741bb8967d0b8b1c15231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51668
I20260620 12:22:32.462431 15915 tablet_service.cc:1514] Processing DeleteTablet for tablet 6727935cf80741bb8967d0b8b1c15231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53002
I20260620 12:22:32.462436 16049 tablet_service.cc:1514] Processing DeleteTablet for tablet 6727935cf80741bb8967d0b8b1c15231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60322
I20260620 12:22:32.462924 16291 tablet_replica.cc:330] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: stopping tablet replica
I20260620 12:22:32.462926 16290 tablet_replica.cc:330] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: stopping tablet replica
I20260620 12:22:32.463107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 15852
I20260620 12:22:32.463255 16290 raft_consensus.cc:2229] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:22:32.464494 16292 tablet_replica.cc:330] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: stopping tablet replica
I20260620 12:22:32.464669 16291 raft_consensus.cc:2229] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:32.464797 16292 raft_consensus.cc:2229] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:32.464980 16291 raft_consensus.cc:2258] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:32.465140 16292 raft_consensus.cc:2258] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:32.466471 16291 ts_tablet_manager.cc:1905] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:32.466943 16292 ts_tablet_manager.cc:1905] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:32.469286 16291 ts_tablet_manager.cc:1918] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:32.469453 16291 log.cc:1196] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/6727935cf80741bb8967d0b8b1c15231
I20260620 12:22:32.469638 16292 ts_tablet_manager.cc:1918] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:32.469733 16292 log.cc:1196] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/6727935cf80741bb8967d0b8b1c15231
I20260620 12:22:32.469838 16291 ts_tablet_manager.cc:1939] T 6727935cf80741bb8967d0b8b1c15231 P 482b78f579214863bfa10951d3c3888e: Deleting consensus metadata
W20260620 12:22:32.469933 15774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44259 (error 108)
I20260620 12:22:32.470079 16292 ts_tablet_manager.cc:1939] T 6727935cf80741bb8967d0b8b1c15231 P 01f32d93042c4468803a462695e906e6: Deleting consensus metadata
I20260620 12:22:32.470180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 15986
I20260620 12:22:32.473426 15774 catalog_manager.cc:4841] TS 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851): tablet 6727935cf80741bb8967d0b8b1c15231 (table test-table [id=564baf07695a4f6fa59d692be66f2078]) successfully deleted
I20260620 12:22:32.473672 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.473297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44259 (local address 127.0.0.1:53024)
0620 12:22:32.473462 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:32.473551 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 12:22:32.473839 15774 catalog_manager.cc:4584] TS d5f59ea98b814bce824ad32365cf2737 (127.0.98.1:44259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6727935cf80741bb8967d0b8b1c15231: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44259: connect: Connection refused (error 111)
I20260620 12:22:32.476071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 16120
I20260620 12:22:32.477571 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:32.476214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.0.1:60332)
0620 12:22:32.477415 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:32.477480 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1103}
I20260620 12:22:32.481594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 15758
I20260620 12:22:52.488118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42469
--webserver_interface=127.0.98.62
--webserver_port=45893
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:22:52.596995 16294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:52.597280 16294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:52.597321 16294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:52.601354 16294 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:22:52.601403 16294 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:22:52.601419 16294 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:22:52.601433 16294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:22:52.601450 16294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:52.601486 16294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:52.601526 16294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:22:52.601549 16294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:22:52.607075 16294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:52.608191 16294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:52.609282 16294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:52.622418 16301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:52.622435 16303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:52.622689 16294 server_base.cc:1034] running on GCE node
W20260620 12:22:52.622435 16300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:52.623355 16294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:52.624526 16294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:52.625723 16294 hybrid_clock.cc:648] HybridClock initialized: now 1781958172625707 us; error 32 us; skew 500 ppm
Jun 20 12:22:52 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:52.633599 16294 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:22:52.634558 16294 webserver.cc:460] Webserver started at http://127.0.98.62:45893/ using document root <none> and password file <none>
I20260620 12:22:52.635159 16294 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:52.635263 16294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:52.639175 16294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:22:52.641451 16310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:52.642604 16294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:52.642773 16294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "88580d6ebedc419aacbf1bf833553ee9"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "4a2dc163cadec2d156158aaad4769e6d"
server_key_iv: "292bfbe05f39138dbae680f92d15c515"
server_key_version: "encryptionkey@0"
I20260620 12:22:52.643380 16294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:52.668607 16294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:52.671995 16294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:52.672212 16294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:52.679306 16294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42469
I20260620 12:22:52.679589 16362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42469 every 8 connection(s)
I20260620 12:22:52.680435 16294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:22:52.684852 16363 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:22:52.684855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 16294
I20260620 12:22:52.685338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44259
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42469
--webserver_port=35331
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:22:52.686636 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Bootstrap starting.
I20260620 12:22:52.703697 16363 log.cc:824] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:52.712107 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:22:52.712487 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Bootstrap complete.
I20260620 12:22:52.716452 16363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:22:52.716809 16363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88580d6ebedc419aacbf1bf833553ee9, State: Initialized, Role: FOLLOWER
I20260620 12:22:52.717317 16363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:22:52.717432 16363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:22:52.717519 16363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:22:52.717630 16363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 12:22:52.719422 16363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:22:52.719743 16363 leader_election.cc:302] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88580d6ebedc419aacbf1bf833553ee9; no voters:
I20260620 12:22:52.720049 16363 leader_election.cc:288] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 12:22:52.720124 16367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 12:22:52.720320 16367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 2 LEADER]: Becoming Leader. State: Replica: 88580d6ebedc419aacbf1bf833553ee9, State: Running, Role: LEADER
I20260620 12:22:52.720649 16367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:22:52.721266 16363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:22:52.721590 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88580d6ebedc419aacbf1bf833553ee9. Latest consensus state: current_term: 2 leader_uuid: "88580d6ebedc419aacbf1bf833553ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } } }
I20260620 12:22:52.721794 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:52.721976 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "88580d6ebedc419aacbf1bf833553ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } } }
I20260620 12:22:52.722150 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:22:52.722543 16371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:22:52.727133 16371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:22:52.727972 16371 catalog_manager.cc:1259] Loaded cluster ID: 1073eac9a25e4b778ca634de9af44951
I20260620 12:22:52.728034 16371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:22:52.730506 16371 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:22:52.731212 16371 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Loaded TSK: 0
I20260620 12:22:52.744019 16371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Generated new TSK 1
I20260620 12:22:52.744252 16371 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:22:52.797741 16365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:52.798017 16365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:52.798086 16365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:52.802142 16365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:52.802224 16365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:52.802315 16365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:22:52.807418 16365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35331
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:52.808442 16365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:52.809511 16365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:52.819820 16393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:52.819856 16390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:52.819857 16391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:52.820300 16365 server_base.cc:1034] running on GCE node
I20260620 12:22:52.820755 16365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:52.821492 16365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:52.822690 16365 hybrid_clock.cc:648] HybridClock initialized: now 1781958172822662 us; error 42 us; skew 500 ppm
Jun 20 12:22:52 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:52.825405 16365 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:22:52.826197 16365 webserver.cc:460] Webserver started at http://127.0.98.1:35331/ using document root <none> and password file <none>
I20260620 12:22:52.826735 16365 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:52.826803 16365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:52.830642 16365 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:22:52.832844 16400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:52.834131 16365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:22:52.834264 16365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "d5f59ea98b814bce824ad32365cf2737"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "2b7e7dba81d5028f4434b19d89221400"
server_key_iv: "45cb0da1c4346e5cbbafda11e4c2defd"
server_key_version: "encryptionkey@0"
I20260620 12:22:52.834810 16365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:52.859498 16365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:52.862818 16365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:52.863027 16365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:52.863641 16365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:52.864739 16407 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:22:52.869601 16365 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:22:52.869699 16365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 12:22:52.869784 16365 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:22:52.871618 16365 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:22:52.871690 16365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 12:22:52.871971 16407 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Bootstrap starting.
I20260620 12:22:52.882256 16365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44259
I20260620 12:22:52.882306 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44259 every 8 connection(s)
I20260620 12:22:52.883176 16365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:22:52.891314 16407 log.cc:824] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:52.892030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 16365
I20260620 12:22:52.892576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35615
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42469
--webserver_port=35917
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:22:52 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:52.899497 16407 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:22:52.900003 16407 tablet_bootstrap.cc:492] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Bootstrap complete.
I20260620 12:22:52.901304 16407 ts_tablet_manager.cc:1397] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.014s
I20260620 12:22:52.902598 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:52.885431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:52827)
0620 12:22:52.886115 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:52.886166 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:52.886965 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:52.888364 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:52.888371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:52.888715 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:52.889252 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:52.889267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:52.891061 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:52.891065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:52.892258 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:52.892269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:52.893281 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:52.893288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:52.893547 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:52.894269 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:52.894285 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:52.896250 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:52.898787 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:52.898796 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:52.898814 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:52.899275 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:52.899761 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:52.899765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:52.899769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:52.899957 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:52.900760 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:52.900785 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:52.901145 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 12:22:52.901357 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 12:22:52.901691 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":158,"threads_started":1}
I20260620 12:22:52.902858 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:52.885386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52827 (local address 127.0.98.62:42469)
0620 12:22:52.885833 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:52.885854 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:52.886323 (+ 469us) server_negotiation.cc:407] Connection header received
0620 12:22:52.887419 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:52.887441 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:52.887936 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:52.888313 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:52.889430 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:52.890888 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:52.892394 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:52.893127 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:52.893292 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:52.896456 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:52.896462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:52.896479 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:52.896513 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:52.898638 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:52.899434 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:52.899445 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:52.899453 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:52.899618 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:52.900087 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:52.900091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:52.900093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:52.900604 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:52.900790 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:52.901509 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 12:22:52.901859 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
I20260620 12:22:52.904199 16407 raft_consensus.cc:348] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:52.904706 16407 raft_consensus.cc:729] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f59ea98b814bce824ad32365cf2737, State: Initialized, Role: FOLLOWER
I20260620 12:22:52.905248 16407 consensus_queue.cc:260] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } } peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260620 12:22:52.905781 16515 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42469
I20260620 12:22:52.905987 16515 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:52.906339 16407 ts_tablet_manager.cc:1428] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:22:52.906481 16515 heartbeater.cc:511] Master 127.0.98.62:42469 requested a full tablet report, sending...
I20260620 12:22:52.908653 16327 ts_manager.cc:194] Registered new tserver with Master: d5f59ea98b814bce824ad32365cf2737 (127.0.98.1:44259)
I20260620 12:22:52.912042 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52827
I20260620 12:22:52.914711 16515 heartbeater.cc:503] Master 127.0.98.62:42469 was elected leader, sending a full tablet report...
Jun 20 12:22:52 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 12:22:52.922555 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:52.911383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44259 (local address 127.0.0.1:55366)
0620 12:22:52.912158 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:52.912199 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:52.912339 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:52.912879 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:52.912884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:52.913268 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:52.913371 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:52.913377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:52.914713 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:52.914718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:52.915564 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:52.915574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:52.916337 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:52.916356 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:52.916443 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:52.916995 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:52.917025 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:52.918655 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:52.920680 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:52.920688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:52.920692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:52.920966 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:52.921287 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:52.921289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:52.921291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:52.921368 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:52.921949 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:52.921955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:52.922071 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 12:22:52.922212 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 12:22:52.922391 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":116,"threads_started":1}
I20260620 12:22:52.922581 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:52.911494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.0.98.1:44259)
0620 12:22:52.911902 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:52.911924 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:52.912295 (+ 371us) server_negotiation.cc:407] Connection header received
0620 12:22:52.912484 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:52.912489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:52.912680 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:52.912817 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:52.913492 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:52.914568 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:52.915702 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:52.916231 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:52.916336 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:52.918818 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:52.918827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:52.918839 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:52.918865 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:52.920564 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:52.921089 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:52.921094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:52.921096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:52.921188 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:52.921457 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:52.921460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:52.921462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:52.921743 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:52.921889 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:52.922337 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 12:22:52.922466 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":111,"threads_started":1}
I20260620 12:22:52.923719 16449 tablet_service.cc:1514] Processing DeleteTablet for tablet 6727935cf80741bb8967d0b8b1c15231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55366
I20260620 12:22:52.924700 16524 tablet_replica.cc:330] stopping tablet replica
I20260620 12:22:52.924997 16524 raft_consensus.cc:2229] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:52.925297 16524 raft_consensus.cc:2258] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:52.926246 16524 ts_tablet_manager.cc:1905] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:52.929714 16524 ts_tablet_manager.cc:1918] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:52.929793 16524 log.cc:1196] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal/wals/6727935cf80741bb8967d0b8b1c15231
I20260620 12:22:52.930094 16524 ts_tablet_manager.cc:1939] T 6727935cf80741bb8967d0b8b1c15231 P d5f59ea98b814bce824ad32365cf2737: Deleting consensus metadata
I20260620 12:22:52.931742 16311 catalog_manager.cc:4841] TS d5f59ea98b814bce824ad32365cf2737 (127.0.98.1:44259): tablet 6727935cf80741bb8967d0b8b1c15231 (table test-table [id=564baf07695a4f6fa59d692be66f2078]) successfully deleted
W20260620 12:22:53.014657 16520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:53.015017 16520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:53.015074 16520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:53.019085 16520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:53.019133 16520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:53.019189 16520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:22:53.024765 16520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35917
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:53.025696 16520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:53.026578 16520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:53.035955 16529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:53.035902 16530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:53.036152 16520 server_base.cc:1034] running on GCE node
W20260620 12:22:53.035902 16532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:53.036832 16520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:53.037668 16520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:53.038839 16520 hybrid_clock.cc:648] HybridClock initialized: now 1781958173038793 us; error 57 us; skew 500 ppm
Jun 20 12:22:53 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:53.043751 16520 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:22:53.045167 16520 webserver.cc:460] Webserver started at http://127.0.98.2:35917/ using document root <none> and password file <none>
I20260620 12:22:53.045945 16520 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:53.046054 16520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:53.052385 16520 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 12:22:53.055365 16539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:53.056806 16520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:22:53.056977 16520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "01f32d93042c4468803a462695e906e6"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "01ed41f037db96cf5294063ad47a1c38"
server_key_iv: "e1f4c00b86d35c39922a0803cd0a242e"
server_key_version: "encryptionkey@0"
I20260620 12:22:53.057881 16520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:53.071455 16520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:53.074399 16520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:53.074600 16520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:53.075142 16520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:53.075920 16520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:53.075994 16520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:53.076064 16520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:53.076116 16520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:53.084532 16520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35615
I20260620 12:22:53.084590 16652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35615 every 8 connection(s)
I20260620 12:22:53.085573 16520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:22:53.091804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 16520
I20260620 12:22:53.092321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41851
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42469
--webserver_port=42075
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 12:22:53 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:53.100803 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.087456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55471 (local address 127.0.98.62:42469)
0620 12:22:53.087594 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.087598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.088432 (+ 834us) server_negotiation.cc:407] Connection header received
0620 12:22:53.089377 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.089380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.089437 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.089541 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:53.090648 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.091185 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.093146 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.093387 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.093471 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.096641 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:53.096647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:53.096652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:53.096693 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:53.098566 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:53.099071 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:53.099078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:53.099082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:53.099181 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:53.099562 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:53.099567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:53.099571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:53.099825 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:53.099940 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.100408 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.100560 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:22:53.101533 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.087679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:55471)
0620 12:22:53.088274 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:53.088325 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:53.089176 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:53.089705 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:53.089712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:53.090057 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:53.090504 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.090517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.091309 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.091312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.092938 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.092952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.093543 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.093549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.093779 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.094524 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:53.094541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:53.096460 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:53.098694 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:53.098699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:53.098709 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:53.098971 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:53.099303 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:53.099306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:53.099308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:53.099458 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:53.099940 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:53.099945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:53.100168 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 12:22:53.100324 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 12:22:53.100599 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":97,"threads_started":1}
I20260620 12:22:53.102783 16653 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42469
I20260620 12:22:53.103029 16653 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:53.103493 16653 heartbeater.cc:511] Master 127.0.98.62:42469 requested a full tablet report, sending...
I20260620 12:22:53.104489 16327 ts_manager.cc:194] Registered new tserver with Master: 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615)
I20260620 12:22:53.105233 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55471
W20260620 12:22:53.202250 16656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:22:53.202490 16656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:22:53.202539 16656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:22:53.206468 16656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:22:53.206521 16656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:22:53.206583 16656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:22:53.211659 16656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42075
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:22:53.212589 16656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:22:53.213582 16656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:22:53.222883 16664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:53.222888 16661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:22:53.223060 16662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:22:53.223253 16656 server_base.cc:1034] running on GCE node
I20260620 12:22:53.223573 16656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:22:53.224200 16656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:22:53.225445 16656 hybrid_clock.cc:648] HybridClock initialized: now 1781958173225424 us; error 32 us; skew 500 ppm
Jun 20 12:22:53 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:22:53.228406 16656 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:22:53.229322 16656 webserver.cc:460] Webserver started at http://127.0.98.3:42075/ using document root <none> and password file <none>
I20260620 12:22:53.229876 16656 fs_manager.cc:359] Metadata directory not provided
I20260620 12:22:53.229945 16656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:22:53.233811 16656 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:22:53.235862 16671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:53.237156 16656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:22:53.237249 16656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "482b78f579214863bfa10951d3c3888e"
format_stamp: "Formatted at 2026-06-20 12:22:32 on dist-test-slave-9b3m"
server_key: "c8431ed9322785f71cc4fe3bac272b62"
server_key_iv: "32f5cacc51633304864f93df07274109"
server_key_version: "encryptionkey@0"
I20260620 12:22:53.237775 16656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:22:53.258849 16656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:22:53.261870 16656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:22:53.262068 16656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:22:53.262667 16656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:22:53.263540 16656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:22:53.263586 16656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:53.263645 16656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:22:53.263677 16656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:22:53.274055 16656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41851
I20260620 12:22:53.274070 16784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41851 every 8 connection(s)
I20260620 12:22:53.275040 16656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:22:53.279222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 16656
Jun 20 12:22:53 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:22:53.287899 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.279917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.98.62:42469)
0620 12:22:53.280171 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.280175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.280265 (+ 90us) server_negotiation.cc:407] Connection header received
0620 12:22:53.280350 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.280353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.280397 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.280491 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:53.281275 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.281797 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.282652 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.282913 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.283023 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.284467 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:53.284470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:53.284473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:53.284496 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:53.286174 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:53.286592 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:53.286598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:53.286603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:53.286691 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:53.286965 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:53.286969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:53.286971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:53.287159 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:53.287367 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.287498 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.287615 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
I20260620 12:22:53.289151 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.276892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52483 (local address 127.0.98.62:42469)
0620 12:22:53.277035 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.277039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.278611 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 12:22:53.279488 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.279492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.279548 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.279629 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:53.281077 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.281612 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.282728 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.282985 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.283101 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.285738 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:53.285741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:53.285742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:53.285773 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:53.287208 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:53.287721 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:53.287725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:53.287726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:53.287801 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:53.288170 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:53.288172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:53.288174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:53.288320 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:53.288395 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.288882 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.288982 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":50}
I20260620 12:22:53.290059 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.277706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:52483)
0620 12:22:53.278420 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:53.278467 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:53.279224 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:53.279800 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:53.279806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:53.280366 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:22:53.280921 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.280933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.281737 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.281741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.282600 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.282608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.283138 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.283142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.283330 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.283845 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:22:53.283860 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:22:53.285557 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:22:53.287343 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:53.287349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:53.287362 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:53.287616 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:53.287890 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:22:53.287896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:22:53.287901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:22:53.288074 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:22:53.288406 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:22:53.288412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:22:53.288655 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 12:22:53.288824 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 12:22:53.289182 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":119,"threads_started":1}
I20260620 12:22:53.291036 16785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42469
I20260620 12:22:53.291285 16785 heartbeater.cc:464] Registering TS with master...
I20260620 12:22:53.291736 16785 heartbeater.cc:511] Master 127.0.98.62:42469 requested a full tablet report, sending...
I20260620 12:22:53.292686 16327 ts_manager.cc:194] Registered new tserver with Master: 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851)
I20260620 12:22:53.293334 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52483
I20260620 12:22:53.301519 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:22:53.305644 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.302091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.0.98.62:42469)
0620 12:22:53.302234 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.302239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.302406 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:22:53.302622 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.302624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.302708 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.302814 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:22:53.303061 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.303555 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.304488 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.304700 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.304809 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.305140 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:22:53.305262 (+ 122us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 12:22:53.305522 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46962: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 12:22:53.305720 16517 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46962: token expired
W20260620 12:22:53.306025 16253 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:42469)
I20260620 12:22:53.313369 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.306389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.0.98.62:42469)
0620 12:22:53.306504 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.306509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.306569 (+ 60us) server_negotiation.cc:407] Connection header received
0620 12:22:53.306678 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.306681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.306723 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.306803 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:22:53.307395 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.307908 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.308699 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.308907 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.309013 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.310305 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:22:53.310309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:22:53.310311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:22:53.310342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:22:53.311735 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:53.312173 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:53.312177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:53.312179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:53.312258 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:22:53.312539 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:22:53.312542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:22:53.312544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:22:53.312706 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:22:53.312839 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.313111 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.313214 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:22:53.315743 16327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:22:53.318974 16327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:22:53.327198 16449 tablet_service.cc:1467] Processing CreateTablet for tablet c125f7cf541f416689607ebecccb7063 (DEFAULT_TABLE table=test-table [id=043c7ce59c1e45089e7b9b2365f6754a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:53.327862 16449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c125f7cf541f416689607ebecccb7063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:53.330669 16407 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Bootstrap starting.
I20260620 12:22:53.331084 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.325696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.0.1:55852)
0620 12:22:53.325862 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:53.325878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:53.325950 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:53.326819 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:53.326822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:53.326844 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:53.326951 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.326959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.328302 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.328305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.329531 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.329542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.330644 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.330647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.330750 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.330795 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 12:22:53.330852 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 12:22:53.330930 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 12:22:53.331248 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.326193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41851 (local address 127.0.0.1:58348)
0620 12:22:53.326438 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:53.326451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:53.326530 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:53.327333 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:53.327336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:53.327353 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:53.327447 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.327454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.328721 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.328727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.329552 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.329561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.330838 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.330843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.330921 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.330939 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:22:53.330993 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 12:22:53.331046 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":7,"thread_start_us":111,"threads_started":1}
I20260620 12:22:53.331552 16407 tablet_bootstrap.cc:654] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:53.331725 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.326394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.0.98.3:41851)
0620 12:22:53.326898 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.326909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.326933 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:22:53.327005 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.327011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.327199 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.327348 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:53.327580 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.328612 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.329701 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.330519 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.330621 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.331467 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.331520 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.331598 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":87,"threads_started":1}
I20260620 12:22:53.331720 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.325886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.0.98.2:35615)
0620 12:22:53.326265 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.326279 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.326304 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:22:53.326401 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.326406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.326639 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.326819 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:53.327063 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.328164 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.329662 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.330500 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.331277 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.331436 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.331492 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.331581 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":146,"threads_started":1}
I20260620 12:22:53.333488 16407 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: No bootstrap required, opened a new log
I20260620 12:22:53.333590 16407 ts_tablet_manager.cc:1397] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 12:22:53.334013 16407 raft_consensus.cc:348] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.334134 16407 raft_consensus.cc:374] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:53.334173 16407 raft_consensus.cc:729] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f59ea98b814bce824ad32365cf2737, State: Initialized, Role: FOLLOWER
I20260620 12:22:53.334296 16407 consensus_queue.cc:260] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.334678 16407 ts_tablet_manager.cc:1428] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 12:22:53.334786 16719 tablet_service.cc:1467] Processing CreateTablet for tablet c125f7cf541f416689607ebecccb7063 (DEFAULT_TABLE table=test-table [id=043c7ce59c1e45089e7b9b2365f6754a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:53.335422 16587 tablet_service.cc:1467] Processing CreateTablet for tablet c125f7cf541f416689607ebecccb7063 (DEFAULT_TABLE table=test-table [id=043c7ce59c1e45089e7b9b2365f6754a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:22:53.335856 16719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c125f7cf541f416689607ebecccb7063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:53.336509 16587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c125f7cf541f416689607ebecccb7063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:22:53.342950 16798 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: Bootstrap starting.
I20260620 12:22:53.343247 16799 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: Bootstrap starting.
I20260620 12:22:53.345268 16798 tablet_bootstrap.cc:654] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:53.345672 16799 tablet_bootstrap.cc:654] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: Neither blocks nor log segments found. Creating new log.
I20260620 12:22:53.346314 16799 log.cc:824] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:53.346315 16798 log.cc:824] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:22:53.348089 16799 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: No bootstrap required, opened a new log
I20260620 12:22:53.348206 16798 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: No bootstrap required, opened a new log
I20260620 12:22:53.348299 16799 ts_tablet_manager.cc:1397] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 12:22:53.348418 16798 ts_tablet_manager.cc:1397] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:22:53.351430 16798 raft_consensus.cc:348] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.351436 16799 raft_consensus.cc:348] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.351686 16798 raft_consensus.cc:374] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:53.351681 16799 raft_consensus.cc:374] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:22:53.351744 16799 raft_consensus.cc:729] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f32d93042c4468803a462695e906e6, State: Initialized, Role: FOLLOWER
I20260620 12:22:53.351745 16798 raft_consensus.cc:729] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482b78f579214863bfa10951d3c3888e, State: Initialized, Role: FOLLOWER
I20260620 12:22:53.352133 16799 consensus_queue.cc:260] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.352162 16798 consensus_queue.cc:260] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.352736 16653 heartbeater.cc:503] Master 127.0.98.62:42469 was elected leader, sending a full tablet report...
I20260620 12:22:53.352751 16785 heartbeater.cc:503] Master 127.0.98.62:42469 was elected leader, sending a full tablet report...
I20260620 12:22:53.353125 16798 ts_tablet_manager.cc:1428] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:22:53.353125 16799 ts_tablet_manager.cc:1428] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260620 12:22:53.384371 16516 tablet.cc:2369] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:22:53.444366 16521 raft_consensus.cc:482] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:22:53.444602 16521 raft_consensus.cc:504] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.445775 16521 leader_election.cc:288] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615), 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851)
I20260620 12:22:53.450090 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.446038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.98.1:54295)
0620 12:22:53.446509 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:53.446520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:53.446696 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:53.446958 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:53.446962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:53.446979 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:53.447080 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.447086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.447863 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.447866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.448901 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.448910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.449602 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.449613 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.449728 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.449763 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 12:22:53.449850 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:22:53.449931 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":242,"thread_start_us":86,"threads_started":1}
I20260620 12:22:53.450068 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.446194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54295 (local address 127.0.98.2:35615)
0620 12:22:53.446386 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.446390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.446600 (+ 210us) server_negotiation.cc:407] Connection header received
0620 12:22:53.446809 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.446817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.446860 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.446943 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:53.447197 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.447743 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.449023 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.449478 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.449596 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.449655 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.449881 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.449951 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 12:22:53.450085 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.446143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49077 (local address 127.0.98.3:41851)
0620 12:22:53.446280 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.446284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.446482 (+ 198us) server_negotiation.cc:407] Connection header received
0620 12:22:53.446692 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.446696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.446748 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.446834 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:22:53.447097 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.447659 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.449028 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.449469 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.449592 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.449653 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.449884 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.449956 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:22:53.450107 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.446038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41851 (local address 127.0.98.1:49077)
0620 12:22:53.446367 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:53.446414 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 12:22:53.446561 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:22:53.446847 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:22:53.446850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:22:53.446866 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:22:53.446991 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.447000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.447772 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.447777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.448900 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:22:53.448909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.449598 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.449601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:22:53.449738 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.449777 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 12:22:53.449868 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 12:22:53.449948 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":86,"thread_start_us":105,"threads_started":1}
I20260620 12:22:53.450745 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c125f7cf541f416689607ebecccb7063" candidate_uuid: "d5f59ea98b814bce824ad32365cf2737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f32d93042c4468803a462695e906e6" is_pre_election: true
I20260620 12:22:53.451092 16607 raft_consensus.cc:2454] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f59ea98b814bce824ad32365cf2737 in term 0.
I20260620 12:22:53.451215 16739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c125f7cf541f416689607ebecccb7063" candidate_uuid: "d5f59ea98b814bce824ad32365cf2737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482b78f579214863bfa10951d3c3888e" is_pre_election: true
I20260620 12:22:53.451536 16739 raft_consensus.cc:2454] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f59ea98b814bce824ad32365cf2737 in term 0.
I20260620 12:22:53.451558 16403 leader_election.cc:302] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f32d93042c4468803a462695e906e6, d5f59ea98b814bce824ad32365cf2737; no voters:
I20260620 12:22:53.451855 16521 raft_consensus.cc:2781] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:22:53.451933 16521 raft_consensus.cc:482] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:22:53.452004 16521 raft_consensus.cc:3037] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:53.453001 16521 raft_consensus.cc:504] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.453451 16521 leader_election.cc:288] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 1 election: Requested vote from peers 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615), 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851)
I20260620 12:22:53.453828 16739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c125f7cf541f416689607ebecccb7063" candidate_uuid: "d5f59ea98b814bce824ad32365cf2737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482b78f579214863bfa10951d3c3888e"
I20260620 12:22:53.453922 16739 raft_consensus.cc:3037] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:53.454838 16739 raft_consensus.cc:2454] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f59ea98b814bce824ad32365cf2737 in term 1.
I20260620 12:22:53.455096 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c125f7cf541f416689607ebecccb7063" candidate_uuid: "d5f59ea98b814bce824ad32365cf2737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f32d93042c4468803a462695e906e6"
I20260620 12:22:53.455144 16401 leader_election.cc:302] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 482b78f579214863bfa10951d3c3888e, d5f59ea98b814bce824ad32365cf2737; no voters:
I20260620 12:22:53.455216 16607 raft_consensus.cc:3037] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:22:53.455335 16521 raft_consensus.cc:2781] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:22:53.455535 16521 raft_consensus.cc:686] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 LEADER]: Becoming Leader. State: Replica: d5f59ea98b814bce824ad32365cf2737, State: Running, Role: LEADER
I20260620 12:22:53.455826 16521 consensus_queue.cc:237] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:22:53.456458 16607 raft_consensus.cc:2454] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f59ea98b814bce824ad32365cf2737 in term 1.
I20260620 12:22:53.458897 16327 catalog_manager.cc:5495] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5f59ea98b814bce824ad32365cf2737 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d5f59ea98b814bce824ad32365cf2737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } health_report { overall_health: HEALTHY } } }
I20260620 12:22:53.486639 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.482675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.0.98.1:44259)
0620 12:22:53.483093 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 12:22:53.483101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:22:53.483116 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:22:53.483194 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:22:53.483197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:22:53.483254 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:22:53.483368 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:22:53.483670 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.484204 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.485138 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:22:53.485372 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:22:53.485491 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:22:53.485772 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:22:53.486198 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:22:53.486330 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:22:53.486392 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 12:22:53.486450 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":150,"threads_started":1}
I20260620 12:22:53.490584 16607 raft_consensus.cc:1261] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 1 FOLLOWER]: Refusing update from remote peer d5f59ea98b814bce824ad32365cf2737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:53.490583 16739 raft_consensus.cc:1261] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 1 FOLLOWER]: Refusing update from remote peer d5f59ea98b814bce824ad32365cf2737: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:22:53.491269 16521 consensus_queue.cc:1035] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:53.491472 16806 consensus_queue.cc:1035] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:22:53.498116 16809 mvcc.cc:205] Tried to move back new op lower bound from 7298900678610325504 to 7298900678477352960. Current Snapshot: MvccSnapshot[applied={T|T < 7298900678610325504}]
I20260620 12:22:53.499018 16811 mvcc.cc:205] Tried to move back new op lower bound from 7298900678610325504 to 7298900678477352960. Current Snapshot: MvccSnapshot[applied={T|T < 7298900678610325504}]
I20260620 12:22:53.499742 16810 mvcc.cc:205] Tried to move back new op lower bound from 7298900678610325504 to 7298900678477352960. Current Snapshot: MvccSnapshot[applied={T|T < 7298900678610325504}]
I20260620 12:22:53.505920 16325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:53.506084 16325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:22:53.508405 16325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Sending DeleteTablet for 3 replicas of tablet c125f7cf541f416689607ebecccb7063
I20260620 12:22:53.509100 16449 tablet_service.cc:1514] Processing DeleteTablet for tablet c125f7cf541f416689607ebecccb7063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55366
I20260620 12:22:53.509173 16587 tablet_service.cc:1514] Processing DeleteTablet for tablet c125f7cf541f416689607ebecccb7063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55852
I20260620 12:22:53.509341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 16365
I20260620 12:22:53.509569 16719 tablet_service.cc:1514] Processing DeleteTablet for tablet c125f7cf541f416689607ebecccb7063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:22:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58348
I20260620 12:22:53.510027 16821 tablet_replica.cc:330] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: stopping tablet replica
I20260620 12:22:53.510331 16821 raft_consensus.cc:2229] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:53.510615 16821 raft_consensus.cc:2258] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:53.511060 16820 tablet_replica.cc:330] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: stopping tablet replica
I20260620 12:22:53.511368 16820 raft_consensus.cc:2229] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:22:53.511641 16820 raft_consensus.cc:2258] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:22:53.513103 16821 ts_tablet_manager.cc:1905] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:53.513100 16820 ts_tablet_manager.cc:1905] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:22:53.515285 16820 ts_tablet_manager.cc:1918] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:53.515396 16820 log.cc:1196] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/c125f7cf541f416689607ebecccb7063
I20260620 12:22:53.515693 16820 ts_tablet_manager.cc:1939] T c125f7cf541f416689607ebecccb7063 P 01f32d93042c4468803a462695e906e6: Deleting consensus metadata
I20260620 12:22:53.516431 16821 ts_tablet_manager.cc:1918] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:22:53.516520 16821 log.cc:1196] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/c125f7cf541f416689607ebecccb7063
I20260620 12:22:53.516928 16313 catalog_manager.cc:4841] TS 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615): tablet c125f7cf541f416689607ebecccb7063 (table test-table [id=043c7ce59c1e45089e7b9b2365f6754a]) successfully deleted
I20260620 12:22:53.517016 16821 ts_tablet_manager.cc:1939] T c125f7cf541f416689607ebecccb7063 P 482b78f579214863bfa10951d3c3888e: Deleting consensus metadata
W20260620 12:22:53.517004 16311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44259 (error 108)
I20260620 12:22:53.517171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 16520
I20260620 12:22:53.518978 16311 catalog_manager.cc:4841] TS 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851): tablet c125f7cf541f416689607ebecccb7063 (table test-table [id=043c7ce59c1e45089e7b9b2365f6754a]) successfully deleted
I20260620 12:22:53.519114 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:22:53.518706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44259 (local address 127.0.0.1:55372)
0620 12:22:53.518889 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 12:22:53.518963 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 12:22:53.519241 16311 catalog_manager.cc:4584] TS d5f59ea98b814bce824ad32365cf2737 (127.0.98.1:44259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c125f7cf541f416689607ebecccb7063: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44259: connect: Connection refused (error 111)
I20260620 12:22:53.523092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 16656
I20260620 12:22:53.528364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 16294
I20260620 12:23:13.534869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42469
--webserver_interface=127.0.98.62
--webserver_port=45893
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:13.645674 16823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:13.645957 16823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:13.646030 16823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:13.650058 16823 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 12:23:13.650122 16823 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 12:23:13.650149 16823 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 12:23:13.650192 16823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:13.650233 16823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:13.650272 16823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:13.650303 16823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:13.650336 16823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:13.655378 16823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:13.656356 16823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:13.657337 16823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:13.665994 16829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:13.665997 16831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:13.665949 16823 server_base.cc:1034] running on GCE node
W20260620 12:23:13.665972 16828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:13.666733 16823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:13.667820 16823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:13.669003 16823 hybrid_clock.cc:648] HybridClock initialized: now 1781958193668970 us; error 42 us; skew 500 ppm
Jun 20 12:23:13 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:13.675482 16823 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:23:13.676306 16823 webserver.cc:460] Webserver started at http://127.0.98.62:45893/ using document root <none> and password file <none>
I20260620 12:23:13.676810 16823 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:13.676867 16823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:13.680588 16823 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:13.682782 16838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:13.684016 16823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:23:13.684214 16823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "88580d6ebedc419aacbf1bf833553ee9"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "4a2dc163cadec2d156158aaad4769e6d"
server_key_iv: "292bfbe05f39138dbae680f92d15c515"
server_key_version: "encryptionkey@0"
I20260620 12:23:13.684751 16823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:13.699121 16823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:13.702145 16823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:13.702363 16823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:13.709599 16823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42469
I20260620 12:23:13.709887 16890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42469 every 8 connection(s)
I20260620 12:23:13.710753 16823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:13.711481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 16823
I20260620 12:23:13.711999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44259
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42469
--webserver_port=35331
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:23:13.715883 16891 sys_catalog.cc:263] Verifying existing consensus state
I20260620 12:23:13.717509 16891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Bootstrap starting.
I20260620 12:23:13.734560 16891 log.cc:824] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:13.745576 16891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:23:13.746021 16891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Bootstrap complete.
I20260620 12:23:13.750123 16891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:23:13.750492 16891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88580d6ebedc419aacbf1bf833553ee9, State: Initialized, Role: FOLLOWER
I20260620 12:23:13.750974 16891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:23:13.751086 16891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:13.751174 16891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:13.751291 16891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 12:23:13.753126 16891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:23:13.753443 16891 leader_election.cc:302] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88580d6ebedc419aacbf1bf833553ee9; no voters:
I20260620 12:23:13.753772 16891 leader_election.cc:288] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 12:23:13.753852 16895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 12:23:13.754060 16895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [term 3 LEADER]: Becoming Leader. State: Replica: 88580d6ebedc419aacbf1bf833553ee9, State: Running, Role: LEADER
I20260620 12:23:13.754344 16895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260620 12:23:13.754995 16891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:13.755203 16896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "88580d6ebedc419aacbf1bf833553ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } } }
I20260620 12:23:13.755241 16897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88580d6ebedc419aacbf1bf833553ee9. Latest consensus state: current_term: 3 leader_uuid: "88580d6ebedc419aacbf1bf833553ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88580d6ebedc419aacbf1bf833553ee9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } } }
I20260620 12:23:13.755321 16896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:13.755355 16897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:13.755677 16899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:13.760705 16899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:13.761418 16899 catalog_manager.cc:1259] Loaded cluster ID: 1073eac9a25e4b778ca634de9af44951
I20260620 12:23:13.761474 16899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:13.763921 16899 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:13.764653 16899 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Loaded TSK: 0
I20260620 12:23:13.764734 16899 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 88580d6ebedc419aacbf1bf833553ee9: Loaded TSK: 1
F20260620 12:23:13.765481 16899 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4a0682b9ad google::LogMessage::Fail() at ??:0
@ 0x7f4a0682e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4a0682b4de google::LogMessage::Flush() at ??:0
@ 0x7f4a0682ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4a094ef18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4a094e7f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4a111907bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4a1116a99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4a1116b178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4a111acdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4a111cb82b std::function<>::operator()() at ??:0
@ 0x7f4a1116ab4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4a1116bf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4a11168579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4a111ac640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4a08a7d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4a08a7dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4a08a7f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4a08a6e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4a0a0b26db start_thread at ??:0
@ 0x7f4a0532961f clone at ??:0
W20260620 12:23:13.841300 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:13.841573 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:13.841620 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:13.845901 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:13.845944 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:13.846001 16893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:23:13.851661 16893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35331
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:13.852715 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:13.853855 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:13.863687 16938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:13.863801 16936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:13.864161 16935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:13.864477 16893 server_base.cc:1034] running on GCE node
I20260620 12:23:13.864867 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:13.865664 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:13.866887 16893 hybrid_clock.cc:648] HybridClock initialized: now 1781958193866853 us; error 44 us; skew 500 ppm
Jun 20 12:23:13 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:13.870254 16893 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:23:13.871341 16893 webserver.cc:460] Webserver started at http://127.0.98.1:35331/ using document root <none> and password file <none>
I20260620 12:23:13.871904 16893 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:13.871961 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:13.876358 16893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 12:23:13.878793 16948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:13.880164 16893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 12:23:13.880338 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "d5f59ea98b814bce824ad32365cf2737"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "2b7e7dba81d5028f4434b19d89221400"
server_key_iv: "45cb0da1c4346e5cbbafda11e4c2defd"
server_key_version: "encryptionkey@0"
I20260620 12:23:13.880892 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:13.894116 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:13.897562 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:13.897765 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:13.898415 16893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:13.899513 16955 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 12:23:13.905097 16893 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 12:23:13.905241 16893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 12:23:13.905332 16893 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 12:23:13.907027 16893 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 12:23:13.907099 16893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 12:23:13.907405 16955 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Bootstrap starting.
I20260620 12:23:13.917943 16893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44259
I20260620 12:23:13.918195 17062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44259 every 8 connection(s)
I20260620 12:23:13.919010 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:23:13.928781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 16893
I20260620 12:23:13.929333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35615
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42469
--webserver_port=35917
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:23:13.929773 16955 log.cc:824] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:13.938675 16955 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 12:23:13.939072 16955 tablet_bootstrap.cc:492] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Bootstrap complete.
I20260620 12:23:13.940505 16955 ts_tablet_manager.cc:1397] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Time spent bootstrapping tablet: real 0.033s user 0.029s sys 0.000s
I20260620 12:23:13.943933 16955 raft_consensus.cc:348] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:23:13.944470 16955 raft_consensus.cc:729] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f59ea98b814bce824ad32365cf2737, State: Initialized, Role: FOLLOWER
I20260620 12:23:13.944991 16955 consensus_queue.cc:260] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:23:13.946105 16955 ts_tablet_manager.cc:1428] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260620 12:23:14.007876 17065 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:23:13.921004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:37615)
0620 12:23:13.921710 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:13.921766 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:13.922708 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:14.007045 (+ 84337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":161,"threads_started":1}
W20260620 12:23:14.008453 17063 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42469 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42469: Client connection negotiation failed: client connection to 127.0.98.62:42469: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 12:23:14.009280 17065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:14.008985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:49947)
0620 12:23:14.009164 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:14.009226 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 12:23:14.010169 17065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:14.009940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:51335)
0620 12:23:14.010071 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:14.010118 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 12:23:14.010479 17063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:23:14.046689 17067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:14.047030 17067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:14.047106 17067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:14.051399 17067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:14.051483 17067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:14.051578 17067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:23:14.056851 17067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35917
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:14.057952 17067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:14.058950 17067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:14.068585 17074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:14.068615 17076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:14.068636 17073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:14.069108 17067 server_base.cc:1034] running on GCE node
I20260620 12:23:14.069579 17067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:14.070326 17067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:14.071550 17067 hybrid_clock.cc:648] HybridClock initialized: now 1781958194071528 us; error 33 us; skew 500 ppm
Jun 20 12:23:14 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:14.074448 17067 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:23:14.075371 17067 webserver.cc:460] Webserver started at http://127.0.98.2:35917/ using document root <none> and password file <none>
I20260620 12:23:14.075924 17067 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:14.076013 17067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:14.079792 17067 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:14.081880 17083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:14.083103 17067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:23:14.083246 17067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "01f32d93042c4468803a462695e906e6"
format_stamp: "Formatted at 2026-06-20 12:22:31 on dist-test-slave-9b3m"
server_key: "01ed41f037db96cf5294063ad47a1c38"
server_key_iv: "e1f4c00b86d35c39922a0803cd0a242e"
server_key_version: "encryptionkey@0"
I20260620 12:23:14.083875 17067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:14.093825 17067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:14.096622 17067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:14.096812 17067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:14.097411 17067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:14.098227 17067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:14.098300 17067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:14.098371 17067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:14.098407 17067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:14.107107 17067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35615
I20260620 12:23:14.107137 17196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35615 every 8 connection(s)
I20260620 12:23:14.108093 17067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:23:14.108327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 17067
I20260620 12:23:14.108875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41851
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42469
--webserver_port=42075
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:23:14.112195 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:14.110516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:49577)
0620 12:23:14.111323 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:14.111456 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":100,"threads_started":1}
W20260620 12:23:14.112643 17197 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42469 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42469: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
I20260620 12:23:14.113255 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:14.113021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:33603)
0620 12:23:14.113142 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:14.113190 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 12:23:14.113867 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:14.113674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:58195)
0620 12:23:14.113767 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:14.113811 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 12:23:14.114048 17197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 12:23:14.227499 17199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:14.227764 17199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:14.227854 17199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:14.235330 17199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:14.235422 17199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:14.235549 17199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:23:14.246132 17199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42075
--tserver_master_addrs=127.0.98.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:14.247614 17199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:14.248947 17199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:14.259559 17206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:14.259559 17205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:14.259708 17199 server_base.cc:1034] running on GCE node
W20260620 12:23:14.259557 17208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:14.260286 17199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:14.260990 17199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:14.262305 17199 hybrid_clock.cc:648] HybridClock initialized: now 1781958194262286 us; error 28 us; skew 500 ppm
Jun 20 12:23:14 dist-test-slave-9b3m krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:14.265360 17199 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:23:14.266265 17199 webserver.cc:460] Webserver started at http://127.0.98.3:42075/ using document root <none> and password file <none>
I20260620 12:23:14.267078 17199 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:14.267222 17199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:14.271324 17199 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 12:23:14.273674 17215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:14.275035 17199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:23:14.275182 17199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "482b78f579214863bfa10951d3c3888e"
format_stamp: "Formatted at 2026-06-20 12:22:32 on dist-test-slave-9b3m"
server_key: "c8431ed9322785f71cc4fe3bac272b62"
server_key_iv: "32f5cacc51633304864f93df07274109"
server_key_version: "encryptionkey@0"
I20260620 12:23:14.275717 17199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:14.291888 17199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:14.295547 17199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:14.295759 17199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:14.296368 17199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:14.297283 17199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:14.297358 17199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:14.297426 17199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:14.297483 17199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:14.307650 17199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41851
I20260620 12:23:14.307685 17328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41851 every 8 connection(s)
I20260620 12:23:14.308657 17199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:23:14.312333 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:14.310770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:58539)
0620 12:23:14.311460 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:14.311568 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":117,"threads_started":1}
W20260620 12:23:14.312885 17329 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42469 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42469: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
I20260620 12:23:14.313486 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:14.313272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:46813)
0620 12:23:14.313381 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:14.313430 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 12:23:14.314105 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:14.313903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:38193)
0620 12:23:14.314016 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:14.314060 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 12:23:14.314306 17329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 12:23:14.315527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 17199
I20260620 12:23:15.011580 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:15.011080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:49105)
0620 12:23:15.011329 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:15.011453 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260620 12:23:15.115020 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:15.114602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:45313)
0620 12:23:15.114854 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:15.114909 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260620 12:23:15.315239 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:15.314859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:42799)
0620 12:23:15.315092 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:15.315144 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
I20260620 12:23:15.333099 17336 raft_consensus.cc:482] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:15.333323 17336 raft_consensus.cc:504] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:23:15.334453 17336 leader_election.cc:288] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615), 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851)
Jun 20 12:23:15 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 12:23:15 dist-test-slave-9b3m krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 12:23:15.348735 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:15.334747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41851 (local address 127.0.98.1:41451)
0620 12:23:15.335123 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:15.335141 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:15.335240 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:15.336898 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:15.336900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:15.337257 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:15.337736 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:15.337750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:15.339002 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:15.339005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:15.339932 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:15.339941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:15.340673 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:15.340677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:15.340928 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:15.341674 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:15.341692 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:15.343690 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:15.346196 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:15.346200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:15.346201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:15.346414 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:15.346750 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:15.346756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:15.346760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:15.346856 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:15.347473 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:15.347480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:15.347892 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 12:23:15.348124 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 12:23:15.348526 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":309,"spinlock_wait_cycles":7040,"thread_start_us":90,"threads_started":1}
I20260620 12:23:15.348783 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:15.334872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37987 (local address 127.0.98.2:35615)
0620 12:23:15.335247 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:15.335267 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:15.335303 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:23:15.335960 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:15.335965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:15.336301 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:15.336539 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:15.337905 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:15.338861 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:15.340065 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:15.340541 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:15.340753 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:15.343593 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:15.343598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:15.343609 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:15.343666 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:15.345753 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:15.346328 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:15.346333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:15.346337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:15.346444 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:15.346827 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:15.346829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:15.346831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:15.347342 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:15.347476 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:15.348314 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 12:23:15.348598 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":83,"threads_started":1}
I20260620 12:23:15.348783 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:15.334923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41451 (local address 127.0.98.3:41851)
0620 12:23:15.335303 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:15.335326 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:15.335371 (+ 45us) server_negotiation.cc:407] Connection header received
0620 12:23:15.335983 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:15.335987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:15.336292 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:15.336649 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:15.337903 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:15.338851 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:15.340057 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:15.340531 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:15.340721 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:15.343864 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:15.343872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:15.343886 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:15.343921 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:15.346083 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:15.346523 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:15.346530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:15.346535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:15.346660 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:15.346938 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:15.346942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:15.346943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:15.347342 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:15.347476 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:15.348314 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 12:23:15.348598 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20260620 12:23:15.348734 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:15.334636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.98.1:37987)
0620 12:23:15.334870 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:15.334884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:15.334988 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:15.336696 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:15.336702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:15.337259 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:15.337738 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:15.337752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:15.339004 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:15.339007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:15.339938 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:15.339947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:15.340667 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:15.340671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:15.340932 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:15.341724 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:15.341736 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:15.343345 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:15.345883 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:15.345887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:15.345898 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:15.346202 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:15.346570 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:15.346573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:15.346575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:15.346733 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:15.347473 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:15.347480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:15.347892 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 12:23:15.348124 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 12:23:15.348569 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":409,"spinlock_wait_cycles":37632}
W20260620 12:23:15.351586 16951 leader_election.cc:341] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615): Not found: Tablet not found: c125f7cf541f416689607ebecccb7063
W20260620 12:23:15.351752 16949 leader_election.cc:341] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851): Not found: Tablet not found: c125f7cf541f416689607ebecccb7063
I20260620 12:23:15.351851 16949 leader_election.cc:302] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f59ea98b814bce824ad32365cf2737; no voters: 01f32d93042c4468803a462695e906e6, 482b78f579214863bfa10951d3c3888e
I20260620 12:23:15.352602 17336 raft_consensus.cc:2726] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:23:16.012997 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:16.012547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:41375)
0620 12:23:16.012841 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:16.012902 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260620 12:23:16.116293 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:16.115897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:35123)
0620 12:23:16.116137 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:16.116188 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260620 12:23:16.316547 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:16.316084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:39671)
0620 12:23:16.316394 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:16.316454 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260620 12:23:17.014400 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:17.013923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:37759)
0620 12:23:17.014238 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:17.014300 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":88,"threads_started":1}
I20260620 12:23:17.117671 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:17.117223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:34407)
0620 12:23:17.117498 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:17.117561 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260620 12:23:17.248557 17345 raft_consensus.cc:482] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:17.248716 17345 raft_consensus.cc:504] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:23:17.249338 17345 leader_election.cc:288] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615), 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851)
W20260620 12:23:17.250365 16949 leader_election.cc:341] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851): Not found: Tablet not found: c125f7cf541f416689607ebecccb7063
W20260620 12:23:17.250463 16951 leader_election.cc:341] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615): Not found: Tablet not found: c125f7cf541f416689607ebecccb7063
I20260620 12:23:17.250592 16951 leader_election.cc:302] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f59ea98b814bce824ad32365cf2737; no voters: 01f32d93042c4468803a462695e906e6, 482b78f579214863bfa10951d3c3888e
I20260620 12:23:17.250783 17345 raft_consensus.cc:2726] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:23:17.318008 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:17.317545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:42697)
0620 12:23:17.317839 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:17.317903 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260620 12:23:18.015674 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:18.015223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:34053)
0620 12:23:18.015493 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:18.015556 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260620 12:23:18.118803 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:18.118424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:47955)
0620 12:23:18.118665 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:18.118718 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260620 12:23:18.319201 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:18.318792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:59653)
0620 12:23:18.319052 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:18.319110 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260620 12:23:19.017282 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:19.016728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:43287)
0620 12:23:19.017096 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:19.017166 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":83,"threads_started":1}
I20260620 12:23:19.120056 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:19.119630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:56003)
0620 12:23:19.119903 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:19.119956 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260620 12:23:19.320371 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:19.319988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:58243)
0620 12:23:19.320230 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:19.320282 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260620 12:23:19.827188 17353 raft_consensus.cc:482] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:19.827354 17353 raft_consensus.cc:504] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:23:19.827765 17353 leader_election.cc:288] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615), 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851)
W20260620 12:23:19.828559 16951 leader_election.cc:341] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615): Not found: Tablet not found: c125f7cf541f416689607ebecccb7063
W20260620 12:23:19.828655 16949 leader_election.cc:341] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851): Not found: Tablet not found: c125f7cf541f416689607ebecccb7063
I20260620 12:23:19.828778 16949 leader_election.cc:302] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f59ea98b814bce824ad32365cf2737; no voters: 01f32d93042c4468803a462695e906e6, 482b78f579214863bfa10951d3c3888e
I20260620 12:23:19.828935 17353 raft_consensus.cc:2726] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:23:20.018720 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:20.018205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:54551)
0620 12:23:20.018504 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:20.018579 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260620 12:23:20.121461 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:20.120801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:58989)
0620 12:23:20.121293 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:20.121363 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260620 12:23:20.321604 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:20.321212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:57417)
0620 12:23:20.321458 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:20.321510 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260620 12:23:21.019935 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:21.019501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:38517)
0620 12:23:21.019779 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:21.019835 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260620 12:23:21.122723 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:21.122268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:53135)
0620 12:23:21.122566 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:21.122627 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":71,"threads_started":1}
I20260620 12:23:21.322757 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:21.322299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:44853)
0620 12:23:21.322609 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:21.322664 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":86,"threads_started":1}
I20260620 12:23:22.021359 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:22.020883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:33799)
0620 12:23:22.021205 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:22.021263 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":126,"threads_started":1}
I20260620 12:23:22.123946 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:22.123569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:48915)
0620 12:23:22.123806 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:22.123858 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260620 12:23:22.323861 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:22.323499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:40783)
0620 12:23:22.323724 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:22.323774 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260620 12:23:23.005466 17363 raft_consensus.cc:482] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:23.005663 17363 raft_consensus.cc:504] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f32d93042c4468803a462695e906e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "482b78f579214863bfa10951d3c3888e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41851 } } peers { permanent_uuid: "d5f59ea98b814bce824ad32365cf2737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44259 } }
I20260620 12:23:23.006211 17363 leader_election.cc:288] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615), 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851)
W20260620 12:23:23.006991 16951 leader_election.cc:341] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01f32d93042c4468803a462695e906e6 (127.0.98.2:35615): Not found: Tablet not found: c125f7cf541f416689607ebecccb7063
W20260620 12:23:23.007114 16949 leader_election.cc:341] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482b78f579214863bfa10951d3c3888e (127.0.98.3:41851): Not found: Tablet not found: c125f7cf541f416689607ebecccb7063
I20260620 12:23:23.007208 16949 leader_election.cc:302] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f59ea98b814bce824ad32365cf2737; no voters: 01f32d93042c4468803a462695e906e6, 482b78f579214863bfa10951d3c3888e
I20260620 12:23:23.007359 17363 raft_consensus.cc:2726] T c125f7cf541f416689607ebecccb7063 P d5f59ea98b814bce824ad32365cf2737 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 12:23:23.022579 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:23.022173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:33291)
0620 12:23:23.022419 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:23.022486 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260620 12:23:23.125144 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:23.124712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:44389)
0620 12:23:23.124949 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:23.124999 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260620 12:23:23.324941 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:23.324565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:53957)
0620 12:23:23.324789 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:23.324838 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260620 12:23:24.023885 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:24.023461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:33063)
0620 12:23:24.023735 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:24.023789 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
I20260620 12:23:24.126171 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:24.125796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:33751)
0620 12:23:24.126016 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:24.126083 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260620 12:23:24.326095 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:24.325673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:46485)
0620 12:23:24.325939 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:24.326005 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":70,"threads_started":1}
I20260620 12:23:25.025143 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:25.024692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:45765)
0620 12:23:25.024940 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:25.024992 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260620 12:23:25.127252 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:25.126839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:43653)
0620 12:23:25.127085 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:25.127138 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
I20260620 12:23:25.327346 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:25.326911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:60603)
0620 12:23:25.327173 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:25.327231 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260620 12:23:26.026283 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:26.025871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:33883)
0620 12:23:26.026127 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:26.026185 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":82,"threads_started":1}
I20260620 12:23:26.128306 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:26.127941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:48805)
0620 12:23:26.128171 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:26.128218 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260620 12:23:26.328588 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:26.328195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:38907)
0620 12:23:26.328441 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:26.328496 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260620 12:23:27.027426 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:27.027014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:60375)
0620 12:23:27.027274 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:27.027331 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260620 12:23:27.129506 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:27.129063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:40661)
0620 12:23:27.129336 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:27.129394 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
I20260620 12:23:27.330013 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:27.329580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:34135)
0620 12:23:27.329832 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:27.329908 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260620 12:23:28.028594 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:28.028191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:53249)
0620 12:23:28.028444 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:28.028498 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260620 12:23:28.130654 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:28.130205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:36907)
0620 12:23:28.130499 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:28.130551 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260620 12:23:28.331120 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:28.330714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:32897)
0620 12:23:28.330971 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:28.331026 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260620 12:23:29.030040 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:29.029585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.1:40993)
0620 12:23:29.029834 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:29.029894 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260620 12:23:29.131807 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:29.131424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.2:40355)
0620 12:23:29.131670 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:29.131718 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260620 12:23:29.319962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 16893
I20260620 12:23:29.325584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 17067
I20260620 12:23:29.332541 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:29.332036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42469 (local address 127.0.98.3:58671)
0620 12:23:29.332315 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:29.332391 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":60,"threads_started":1}
I20260620 12:23:29.332832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 17199
2026-06-20T12:23:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59258 ms)
[----------] 30 tests from SecurityITest (241439 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:29 dist-test-slave-9b3m krb5kdc[17387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:29 dist-test-slave-9b3m krb5kdc[17387](info): set up 2 sockets
Jun 20 12:23:29 dist-test-slave-9b3m krb5kdc[17387](info): commencing operation
krb5kdc: starting...
W20260620 12:23:30.380805 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:30 dist-test-slave-9b3m krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:23:30.500157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:30.608659 17400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:30.608937 17400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:30.609014 17400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:30.612900 17400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:30.612979 17400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:30.613015 17400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:30.613088 17400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:30.613138 17400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:30.618393 17400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:30.619388 17400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:30.620389 17400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:30.629068 17408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:30.629118 17405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:30.629120 17406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:30.629182 17400 server_base.cc:1034] running on GCE node
I20260620 12:23:30.629875 17400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:30.630898 17400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:30.632057 17400 hybrid_clock.cc:648] HybridClock initialized: now 1781958210632022 us; error 47 us; skew 500 ppm
Jun 20 12:23:30 dist-test-slave-9b3m krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:30.634968 17400 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:23:30.635859 17400 webserver.cc:460] Webserver started at http://127.0.98.62:37039/ using document root <none> and password file <none>
I20260620 12:23:30.636360 17400 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:30.636418 17400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:30.636574 17400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:30.638566 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "4dcaa9ecff4244b2baf45f56bb0570b5"
format_stamp: "Formatted at 2026-06-20 12:23:30 on dist-test-slave-9b3m"
server_key: "212aed7b1f61390e4e24b87a09396bb5"
server_key_iv: "1be9e03dd070b9b4efd26793a0cc7dca"
server_key_version: "encryptionkey@0"
I20260620 12:23:30.639065 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "4dcaa9ecff4244b2baf45f56bb0570b5"
format_stamp: "Formatted at 2026-06-20 12:23:30 on dist-test-slave-9b3m"
server_key: "212aed7b1f61390e4e24b87a09396bb5"
server_key_iv: "1be9e03dd070b9b4efd26793a0cc7dca"
server_key_version: "encryptionkey@0"
I20260620 12:23:30.642593 17400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 12:23:30.644862 17415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:30.646133 17400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 12:23:30.646270 17400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "4dcaa9ecff4244b2baf45f56bb0570b5"
format_stamp: "Formatted at 2026-06-20 12:23:30 on dist-test-slave-9b3m"
server_key: "212aed7b1f61390e4e24b87a09396bb5"
server_key_iv: "1be9e03dd070b9b4efd26793a0cc7dca"
server_key_version: "encryptionkey@0"
I20260620 12:23:30.646381 17400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:30.662156 17400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:30.664886 17400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:30.665153 17400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:30.672302 17400 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46795
I20260620 12:23:30.672315 17467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46795 every 8 connection(s)
I20260620 12:23:30.673290 17400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:30.675421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 17400
I20260620 12:23:30.675572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:30.675865 392 external_mini_cluster.cc:1428] Setting key 0b00c751354b1324640e92502313419f
I20260620 12:23:30.676352 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:30.683403 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5: Bootstrap starting.
Jun 20 12:23:30 dist-test-slave-9b3m krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:30.685650 17468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:30.686570 17468 log.cc:824] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:30.688683 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5: No bootstrap required, opened a new log
I20260620 12:23:30.690790 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:30.677286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.0.98.62:46795)
0620 12:23:30.677781 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:30.677802 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:30.677842 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:23:30.678773 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:30.678798 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:30.679208 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:30.679569 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:30.680206 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:30.681688 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:30.682530 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:30.683184 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:30.683335 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:30.685888 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:30.685896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:30.685915 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:30.685950 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:30.687805 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:30.688200 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:30.688205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:30.688210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:30.688338 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:30.688583 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:30.688586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:30.688588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:30.689076 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:30.689278 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:30.689592 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 12:23:30.689868 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":159,"threads_started":1}
I20260620 12:23:30.692916 17468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcaa9ecff4244b2baf45f56bb0570b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260620 12:23:30.693185 17468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:30.693225 17468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dcaa9ecff4244b2baf45f56bb0570b5, State: Initialized, Role: FOLLOWER
I20260620 12:23:30.693557 17468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcaa9ecff4244b2baf45f56bb0570b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260620 12:23:30.693714 17468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:30.693759 17468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:30.693825 17468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:30.694587 17468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcaa9ecff4244b2baf45f56bb0570b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260620 12:23:30.694859 17468 leader_election.cc:302] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dcaa9ecff4244b2baf45f56bb0570b5; no voters:
I20260620 12:23:30.695082 17468 leader_election.cc:288] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:30.695214 17473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:30.695474 17473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [term 1 LEADER]: Becoming Leader. State: Replica: 4dcaa9ecff4244b2baf45f56bb0570b5, State: Running, Role: LEADER
I20260620 12:23:30.695781 17473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcaa9ecff4244b2baf45f56bb0570b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260620 12:23:30.696221 17468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:30.697357 17475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dcaa9ecff4244b2baf45f56bb0570b5. Latest consensus state: current_term: 1 leader_uuid: "4dcaa9ecff4244b2baf45f56bb0570b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcaa9ecff4244b2baf45f56bb0570b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } } }
I20260620 12:23:30.697495 17475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:30.697880 17482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:30.697855 17474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dcaa9ecff4244b2baf45f56bb0570b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcaa9ecff4244b2baf45f56bb0570b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } } }
I20260620 12:23:30.697986 17474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:30.701104 17482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:30.706426 17482 catalog_manager.cc:1347] Generated new cluster ID: d35a327dd1e7472d9f64dfb434860a8e
I20260620 12:23:30.706488 17482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:30.717602 17482 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:30.718473 17482 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:30.727129 17482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5: Generated new TSK 0
I20260620 12:23:30.727752 17482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:23:30.753786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:30.862011 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:30.862250 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:30.862290 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:30.866230 17494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:30.866279 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:30.866364 17494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:23:30.871425 17494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:30.872398 17494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:30.873416 17494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:30.882393 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:30.882401 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:30.882490 17494 server_base.cc:1034] running on GCE node
W20260620 12:23:30.882450 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:30.882984 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:30.883621 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:30.884809 17494 hybrid_clock.cc:648] HybridClock initialized: now 1781958210884775 us; error 45 us; skew 500 ppm
Jun 20 12:23:30 dist-test-slave-9b3m krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:30.887485 17494 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:23:30.888302 17494 webserver.cc:460] Webserver started at http://127.0.98.1:35151/ using document root <none> and password file <none>
I20260620 12:23:30.888846 17494 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:30.888934 17494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:30.889225 17494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:30.891235 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "90f5c5540d124e7f8ddf607a3e5601be"
format_stamp: "Formatted at 2026-06-20 12:23:30 on dist-test-slave-9b3m"
server_key: "060b0418255402669085e31cfd695bfe"
server_key_iv: "759aba311528e1879a0a73ea70578889"
server_key_version: "encryptionkey@0"
I20260620 12:23:30.891789 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "90f5c5540d124e7f8ddf607a3e5601be"
format_stamp: "Formatted at 2026-06-20 12:23:30 on dist-test-slave-9b3m"
server_key: "060b0418255402669085e31cfd695bfe"
server_key_iv: "759aba311528e1879a0a73ea70578889"
server_key_version: "encryptionkey@0"
I20260620 12:23:30.895392 17494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:23:30.898069 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:30.899217 17494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:30.899345 17494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "90f5c5540d124e7f8ddf607a3e5601be"
format_stamp: "Formatted at 2026-06-20 12:23:30 on dist-test-slave-9b3m"
server_key: "060b0418255402669085e31cfd695bfe"
server_key_iv: "759aba311528e1879a0a73ea70578889"
server_key_version: "encryptionkey@0"
I20260620 12:23:30.899451 17494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:30.919509 17494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:30.922238 17494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:30.922437 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:30.923002 17494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:30.923811 17494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:30.923880 17494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:30.923949 17494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:30.923997 17494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:30.932880 17494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43995
I20260620 12:23:30.932909 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43995 every 8 connection(s)
I20260620 12:23:30.933755 17494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:23:30.939679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 17494
I20260620 12:23:30.939831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:23:30.940105 392 external_mini_cluster.cc:1428] Setting key 2c212e320f7e284cbaafc936d74371d4
Jun 20 12:23:30 dist-test-slave-9b3m krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:30.947997 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:30.935511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41039 (local address 127.0.98.62:46795)
0620 12:23:30.935675 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:30.935680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:30.936488 (+ 808us) server_negotiation.cc:407] Connection header received
0620 12:23:30.937518 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:30.937522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:30.937584 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:30.937703 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:30.938881 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:30.939494 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:30.941014 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:30.941325 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:30.941449 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:30.943995 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:30.944001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:30.944006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:30.944041 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:30.945685 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:30.946352 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:30.946359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:30.946364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:30.946460 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:30.946886 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:30.946889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:30.946890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:30.947067 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:30.947156 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:30.947770 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 12:23:30.947872 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:23:30.949131 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:30.935887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.1:41039)
0620 12:23:30.936335 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:30.936377 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:30.937291 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:30.937877 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:30.937884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:30.938244 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:30.938723 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:30.938737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:30.939654 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:30.939657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:30.940884 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:30.940894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:30.941482 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:30.941487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:30.941678 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:30.942254 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:30.942270 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:30.943824 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:30.945808 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:30.945814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:30.945828 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:30.946196 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:30.946570 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:30.946574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:30.946578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:30.946780 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:30.947176 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:30.947184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:30.947497 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 12:23:30.947724 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 12:23:30.948065 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:23:30.950420 17623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260620 12:23:30.950695 17623 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:30.951146 17623 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:23:30.952813 17432 ts_manager.cc:194] Registered new tserver with Master: 90f5c5540d124e7f8ddf607a3e5601be (127.0.98.1:43995)
I20260620 12:23:30.954468 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:23:30.967900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:31.075312 17628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:31.075573 17628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:31.075623 17628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:31.079635 17628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:31.079687 17628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:31.079787 17628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:23:31.084893 17628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:31.085995 17628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:31.086988 17628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:31.096514 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:31.096520 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:31.096520 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:31.096966 17628 server_base.cc:1034] running on GCE node
I20260620 12:23:31.097504 17628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:31.098188 17628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:31.099359 17628 hybrid_clock.cc:648] HybridClock initialized: now 1781958211099331 us; error 51 us; skew 500 ppm
Jun 20 12:23:31 dist-test-slave-9b3m krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:31.102092 17628 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:23:31.102908 17628 webserver.cc:460] Webserver started at http://127.0.98.2:45263/ using document root <none> and password file <none>
I20260620 12:23:31.103467 17628 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:31.103524 17628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:31.103681 17628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:31.105715 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "97408e06e59a495894e71b51d054ec38"
format_stamp: "Formatted at 2026-06-20 12:23:31 on dist-test-slave-9b3m"
server_key: "dee784761908e2d681da1408fa85bad3"
server_key_iv: "b7d34b72248843a85d7106d6da5d4c2b"
server_key_version: "encryptionkey@0"
I20260620 12:23:31.106217 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97408e06e59a495894e71b51d054ec38"
format_stamp: "Formatted at 2026-06-20 12:23:31 on dist-test-slave-9b3m"
server_key: "dee784761908e2d681da1408fa85bad3"
server_key_iv: "b7d34b72248843a85d7106d6da5d4c2b"
server_key_version: "encryptionkey@0"
I20260620 12:23:31.109901 17628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:23:31.112264 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:31.113628 17628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:31.113744 17628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "97408e06e59a495894e71b51d054ec38"
format_stamp: "Formatted at 2026-06-20 12:23:31 on dist-test-slave-9b3m"
server_key: "dee784761908e2d681da1408fa85bad3"
server_key_iv: "b7d34b72248843a85d7106d6da5d4c2b"
server_key_version: "encryptionkey@0"
I20260620 12:23:31.113829 17628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:31.141304 17628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:31.144308 17628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:31.144544 17628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:31.145197 17628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:31.146019 17628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:31.146090 17628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:31.146163 17628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:31.146234 17628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:31.155198 17628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33961
I20260620 12:23:31.155233 17756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33961 every 8 connection(s)
I20260620 12:23:31.156090 17628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:23:31.163980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 17628
I20260620 12:23:31.164140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:23:31.164444 392 external_mini_cluster.cc:1428] Setting key f4cdae5c3322c8fcabf03e22d0af90f9
Jun 20 12:23:31 dist-test-slave-9b3m krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:31.169658 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.157872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58265 (local address 127.0.98.62:46795)
0620 12:23:31.158003 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.158007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.158837 (+ 830us) server_negotiation.cc:407] Connection header received
0620 12:23:31.159727 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.159730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.159798 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.159902 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:31.161080 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.161730 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.162863 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.163091 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.163203 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.165888 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:31.165892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:31.165894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:31.165923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:31.167381 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:31.167980 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:31.167985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:31.167989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:31.168086 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:31.168408 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:31.168413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:31.168416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:31.168654 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:31.168802 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.169272 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.169412 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:23:31.170444 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.158260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.2:58265)
0620 12:23:31.158686 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:31.158728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:31.159525 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:31.160066 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:31.160072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:31.160431 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:31.160894 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.160907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.161877 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.161879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.162737 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.162747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.163248 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.163252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.163417 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.164017 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:31.164057 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:31.165734 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:31.167490 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:31.167499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:31.167517 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:31.167883 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:31.168186 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:31.168189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:31.168190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:31.168317 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:31.168764 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:31.168768 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:31.168983 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 12:23:31.169194 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 12:23:31.169439 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260620 12:23:31.171631 17757 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260620 12:23:31.171924 17757 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:31.172510 17757 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
I20260620 12:23:31.173750 17432 ts_manager.cc:194] Registered new tserver with Master: 97408e06e59a495894e71b51d054ec38 (127.0.98.2:33961)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:23:31.174360 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:23:31.192811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:31.302975 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:31.303268 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:31.303368 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:31.307443 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:31.307524 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:31.307611 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:23:31.312762 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:31.313798 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:31.314798 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:31.324458 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:31.324542 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:31.324543 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:31.324728 17762 server_base.cc:1034] running on GCE node
I20260620 12:23:31.325240 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:31.325904 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:31.327088 17762 hybrid_clock.cc:648] HybridClock initialized: now 1781958211327072 us; error 28 us; skew 500 ppm
Jun 20 12:23:31 dist-test-slave-9b3m krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:31.329998 17762 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:23:31.330845 17762 webserver.cc:460] Webserver started at http://127.0.98.3:34857/ using document root <none> and password file <none>
I20260620 12:23:31.331399 17762 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:31.331489 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:31.331718 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:31.333783 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "7a3205511f01452f93c78874d25da9d9"
format_stamp: "Formatted at 2026-06-20 12:23:31 on dist-test-slave-9b3m"
server_key: "f4f852905b19203a64c1584e58077a14"
server_key_iv: "a8a88047c6653276e3ad5347d8eb0bbd"
server_key_version: "encryptionkey@0"
I20260620 12:23:31.334328 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7a3205511f01452f93c78874d25da9d9"
format_stamp: "Formatted at 2026-06-20 12:23:31 on dist-test-slave-9b3m"
server_key: "f4f852905b19203a64c1584e58077a14"
server_key_iv: "a8a88047c6653276e3ad5347d8eb0bbd"
server_key_version: "encryptionkey@0"
I20260620 12:23:31.337949 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:23:31.340233 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:31.341464 17762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:31.341591 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "7a3205511f01452f93c78874d25da9d9"
format_stamp: "Formatted at 2026-06-20 12:23:31 on dist-test-slave-9b3m"
server_key: "f4f852905b19203a64c1584e58077a14"
server_key_iv: "a8a88047c6653276e3ad5347d8eb0bbd"
server_key_version: "encryptionkey@0"
I20260620 12:23:31.341694 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:31.369561 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:31.372467 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:31.372671 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:31.373270 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:31.374070 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:31.374140 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:31.374210 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:31.374253 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:31.383141 17762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38553
I20260620 12:23:31.383188 17890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38553 every 8 connection(s)
I20260620 12:23:31.384034 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:23:31.388856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 17762
I20260620 12:23:31.389001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:23:31.389369 392 external_mini_cluster.cc:1428] Setting key ded278ba71330a104eeb7264722d503e
Jun 20 12:23:31 dist-test-slave-9b3m krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:31.397795 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.385907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55033 (local address 127.0.98.62:46795)
0620 12:23:31.386080 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.386086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.386991 (+ 905us) server_negotiation.cc:407] Connection header received
0620 12:23:31.387798 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.387801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.387847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.387919 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:31.389275 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.389964 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.391551 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.391785 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.391896 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.394395 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:31.394398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:31.394400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:31.394424 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:31.395640 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:31.396165 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:31.396170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:31.396172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:31.396256 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:31.396593 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:31.396596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:31.396598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:31.396866 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:31.396997 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.397502 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.397620 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 12:23:31.398592 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.386323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.3:55033)
0620 12:23:31.386835 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:31.386877 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:31.387605 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:31.388070 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:31.388076 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:31.388422 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:31.389121 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.389142 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.390087 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.390090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.391438 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.391449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.391932 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.391936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.392114 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.392722 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:31.392739 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:31.394234 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:31.395791 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:31.395797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:31.395811 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:31.396061 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:31.396348 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:31.396352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:31.396354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:31.396490 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:31.396986 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:31.396993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:31.397272 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 12:23:31.397461 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 12:23:31.397765 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":111,"threads_started":1}
I20260620 12:23:31.399585 17891 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260620 12:23:31.399822 17891 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:31.400298 17891 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
I20260620 12:23:31.401296 17432 ts_manager.cc:194] Registered new tserver with Master: 7a3205511f01452f93c78874d25da9d9 (127.0.98.3:38553)
I20260620 12:23:31.402016 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55033
I20260620 12:23:31.403106 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:23:31.413107 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.405093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.62:46795)
0620 12:23:31.405207 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.405211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.405451 (+ 240us) server_negotiation.cc:407] Connection header received
0620 12:23:31.405593 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.405597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.405636 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.405726 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:31.406391 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.406977 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.407970 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.408220 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.408316 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.409750 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:31.409756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:31.409759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:31.409789 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:31.411457 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:31.411878 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:31.411883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:31.411885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:31.411967 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:31.412232 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:31.412237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:31.412238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:31.412447 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:31.412584 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.412815 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.412923 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:23:31.415580 17432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:23:31.418890 17432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:23:31.432840 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.426912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43995 (local address 127.0.0.1:42992)
0620 12:23:31.427720 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:31.427735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:31.427823 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:31.428561 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:31.428563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:31.428580 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:31.428680 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.428686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.430065 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.430068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.431367 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.431378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.432304 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.432308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.432414 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.432457 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 12:23:31.432558 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 12:23:31.432672 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":57,"thread_start_us":92,"threads_started":1}
I20260620 12:23:31.433112 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.427650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.0.98.2:33961)
0620 12:23:31.428054 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.428066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.428406 (+ 340us) server_negotiation.cc:407] Connection header received
0620 12:23:31.428484 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.428490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.428667 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.428802 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:31.429210 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.430217 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.431845 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.432431 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.432550 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.432692 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.432793 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.432898 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":105,"threads_started":1}
I20260620 12:23:31.433022 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.427426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33961 (local address 127.0.0.1:33110)
0620 12:23:31.428211 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:31.428225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:31.428305 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:31.428962 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:31.428965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:31.428982 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:31.429089 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.429095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.430531 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.430535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.431698 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.431708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.432554 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.432558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.432647 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.432671 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:23:31.432756 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 12:23:31.432821 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":90,"spinlock_wait_cycles":384,"thread_start_us":92,"threads_started":1}
I20260620 12:23:31.433436 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.427497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.0.98.1:43995)
0620 12:23:31.428107 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.428124 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.428145 (+ 21us) server_negotiation.cc:407] Connection header received
0620 12:23:31.428218 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.428223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.428424 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.428587 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:31.428801 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.429934 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.431494 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.432188 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.432281 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.432385 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.433172 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.433280 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":70,"threads_started":1}
I20260620 12:23:31.434363 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.426709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38553 (local address 127.0.0.1:41814)
0620 12:23:31.427114 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:31.427177 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:31.427307 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:31.428233 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:31.428242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:31.428271 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:31.428432 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.428439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.429833 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.429839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.431384 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.431390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.433953 (+ 2563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.433957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.434042 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.434066 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 12:23:31.434156 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:23:31.434233 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":115,"spinlock_wait_cycles":6272,"thread_start_us":80,"threads_started":1}
I20260620 12:23:31.434403 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.426852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.0.98.3:38553)
0620 12:23:31.427309 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.427321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.427346 (+ 25us) server_negotiation.cc:407] Connection header received
0620 12:23:31.427825 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.427831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.428042 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.428575 (+ 533us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:31.428747 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.429702 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.432433 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.433833 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.433941 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.434082 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.434189 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.434292 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":153,"threads_started":1}
I20260620 12:23:31.436645 17825 tablet_service.cc:1467] Processing CreateTablet for tablet 20da48f2eb964385a56b281987ddef33 (DEFAULT_TABLE table=test-table [id=7b81189b3841436f827438e45baf410d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:31.436780 17557 tablet_service.cc:1467] Processing CreateTablet for tablet 20da48f2eb964385a56b281987ddef33 (DEFAULT_TABLE table=test-table [id=7b81189b3841436f827438e45baf410d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:31.436775 17691 tablet_service.cc:1467] Processing CreateTablet for tablet 20da48f2eb964385a56b281987ddef33 (DEFAULT_TABLE table=test-table [id=7b81189b3841436f827438e45baf410d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:31.437745 17825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20da48f2eb964385a56b281987ddef33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:31.437789 17557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20da48f2eb964385a56b281987ddef33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:31.437789 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20da48f2eb964385a56b281987ddef33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:31.443730 17908 tablet_bootstrap.cc:492] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Bootstrap starting.
I20260620 12:23:31.445384 17909 tablet_bootstrap.cc:492] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be: Bootstrap starting.
I20260620 12:23:31.446461 17908 tablet_bootstrap.cc:654] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:31.447302 17909 tablet_bootstrap.cc:654] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:31.447570 17908 log.cc:824] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:31.448313 17909 log.cc:824] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:31.448890 17910 tablet_bootstrap.cc:492] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Bootstrap starting.
I20260620 12:23:31.449606 17908 tablet_bootstrap.cc:492] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: No bootstrap required, opened a new log
I20260620 12:23:31.449817 17908 ts_tablet_manager.cc:1397] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:23:31.450623 17910 tablet_bootstrap.cc:654] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:31.451474 17910 log.cc:824] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:31.453130 17909 tablet_bootstrap.cc:492] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be: No bootstrap required, opened a new log
I20260620 12:23:31.453130 17910 tablet_bootstrap.cc:492] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: No bootstrap required, opened a new log
I20260620 12:23:31.453315 17909 ts_tablet_manager.cc:1397] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 12:23:31.453315 17910 ts_tablet_manager.cc:1397] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:23:31.453258 17908 raft_consensus.cc:348] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.453549 17908 raft_consensus.cc:374] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:31.453620 17908 raft_consensus.cc:729] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3205511f01452f93c78874d25da9d9, State: Initialized, Role: FOLLOWER
I20260620 12:23:31.454071 17908 consensus_queue.cc:260] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.454737 17891 heartbeater.cc:503] Master 127.0.98.62:46795 was elected leader, sending a full tablet report...
I20260620 12:23:31.455039 17908 ts_tablet_manager.cc:1428] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:23:31.457019 17910 raft_consensus.cc:348] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.457327 17910 raft_consensus.cc:374] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:31.457190 17909 raft_consensus.cc:348] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.457422 17909 raft_consensus.cc:374] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:31.457443 17910 raft_consensus.cc:729] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97408e06e59a495894e71b51d054ec38, State: Initialized, Role: FOLLOWER
I20260620 12:23:31.457502 17909 raft_consensus.cc:729] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f5c5540d124e7f8ddf607a3e5601be, State: Initialized, Role: FOLLOWER
I20260620 12:23:31.457881 17910 consensus_queue.cc:260] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.457942 17909 consensus_queue.cc:260] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.458537 17757 heartbeater.cc:503] Master 127.0.98.62:46795 was elected leader, sending a full tablet report...
I20260620 12:23:31.458817 17910 ts_tablet_manager.cc:1428] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 12:23:31.458827 17909 ts_tablet_manager.cc:1428] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:23:31.459200 17623 heartbeater.cc:503] Master 127.0.98.62:46795 was elected leader, sending a full tablet report...
W20260620 12:23:31.635030 17892 tablet.cc:2369] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:23:31.657240 17758 tablet.cc:2369] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:23:31.685009 17624 tablet.cc:2369] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:31.735958 17915 raft_consensus.cc:482] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:31.736169 17915 raft_consensus.cc:504] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.737427 17915 leader_election.cc:288] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a3205511f01452f93c78874d25da9d9 (127.0.98.3:38553), 90f5c5540d124e7f8ddf607a3e5601be (127.0.98.1:43995)
I20260620 12:23:31.741277 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.737860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51063 (local address 127.0.98.1:43995)
0620 12:23:31.737996 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.738001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.738160 (+ 159us) server_negotiation.cc:407] Connection header received
0620 12:23:31.738336 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.738338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.738378 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.738458 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:31.738739 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.739230 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.740271 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.740648 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.740766 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.740830 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.741061 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.741134 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 12:23:31.741277 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.737755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34251 (local address 127.0.98.3:38553)
0620 12:23:31.737893 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.737898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.738065 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:23:31.738232 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.738239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.738287 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.738369 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:31.738643 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.739140 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.740259 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.740657 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.740772 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.740833 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.741073 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.741153 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 12:23:31.741312 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.737722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43995 (local address 127.0.98.2:51063)
0620 12:23:31.738052 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:31.738070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:31.738202 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:31.738480 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:31.738483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:31.738498 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:31.738628 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.738633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.739335 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.739340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.740145 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.740154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.740767 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.740771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.740900 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.740927 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:23:31.741014 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:23:31.741133 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":29,"thread_start_us":62,"threads_started":1}
I20260620 12:23:31.741317 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.737613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38553 (local address 127.0.98.2:34251)
0620 12:23:31.737963 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:31.737975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:31.738112 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:31.738380 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:31.738383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:31.738398 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:31.738526 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.738533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.739247 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.739251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.740133 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:31.740142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.740775 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.740779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:31.740930 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.740948 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:23:31.741035 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:23:31.741161 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":86,"thread_start_us":102,"threads_started":1}
I20260620 12:23:31.742035 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20da48f2eb964385a56b281987ddef33" candidate_uuid: "97408e06e59a495894e71b51d054ec38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f5c5540d124e7f8ddf607a3e5601be" is_pre_election: true
I20260620 12:23:31.742111 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20da48f2eb964385a56b281987ddef33" candidate_uuid: "97408e06e59a495894e71b51d054ec38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3205511f01452f93c78874d25da9d9" is_pre_election: true
I20260620 12:23:31.742363 17577 raft_consensus.cc:2454] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97408e06e59a495894e71b51d054ec38 in term 0.
I20260620 12:23:31.742401 17845 raft_consensus.cc:2454] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97408e06e59a495894e71b51d054ec38 in term 0.
I20260620 12:23:31.742844 17646 leader_election.cc:302] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a3205511f01452f93c78874d25da9d9, 97408e06e59a495894e71b51d054ec38; no voters:
I20260620 12:23:31.743122 17915 raft_consensus.cc:2781] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:23:31.743197 17915 raft_consensus.cc:482] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:23:31.743242 17915 raft_consensus.cc:3037] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:31.744266 17915 raft_consensus.cc:504] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.744639 17915 leader_election.cc:288] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [CANDIDATE]: Term 1 election: Requested vote from peers 7a3205511f01452f93c78874d25da9d9 (127.0.98.3:38553), 90f5c5540d124e7f8ddf607a3e5601be (127.0.98.1:43995)
I20260620 12:23:31.745069 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20da48f2eb964385a56b281987ddef33" candidate_uuid: "97408e06e59a495894e71b51d054ec38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3205511f01452f93c78874d25da9d9"
I20260620 12:23:31.745064 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20da48f2eb964385a56b281987ddef33" candidate_uuid: "97408e06e59a495894e71b51d054ec38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f5c5540d124e7f8ddf607a3e5601be"
I20260620 12:23:31.745185 17845 raft_consensus.cc:3037] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:31.745185 17577 raft_consensus.cc:3037] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:31.746469 17577 raft_consensus.cc:2454] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97408e06e59a495894e71b51d054ec38 in term 1.
I20260620 12:23:31.746469 17845 raft_consensus.cc:2454] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97408e06e59a495894e71b51d054ec38 in term 1.
I20260620 12:23:31.746817 17647 leader_election.cc:302] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f5c5540d124e7f8ddf607a3e5601be, 97408e06e59a495894e71b51d054ec38; no voters:
I20260620 12:23:31.747015 17915 raft_consensus.cc:2781] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:31.747185 17915 raft_consensus.cc:686] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 1 LEADER]: Becoming Leader. State: Replica: 97408e06e59a495894e71b51d054ec38, State: Running, Role: LEADER
I20260620 12:23:31.747495 17915 consensus_queue.cc:237] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } }
I20260620 12:23:31.750597 17431 catalog_manager.cc:5495] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97408e06e59a495894e71b51d054ec38 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "97408e06e59a495894e71b51d054ec38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97408e06e59a495894e71b51d054ec38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33961 } health_report { overall_health: HEALTHY } } }
I20260620 12:23:31.799345 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.795560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.0.98.2:33961)
0620 12:23:31.795725 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:31.795729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:31.795744 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:23:31.795944 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:31.795947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:31.796003 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:31.796114 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:23:31.796363 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.796957 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.797839 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:31.798133 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:31.798244 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:31.798548 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:23:31.798945 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:23:31.799074 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:31.799136 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 12:23:31.799191 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:23:31.805140 17577 raft_consensus.cc:1261] T 20da48f2eb964385a56b281987ddef33 P 90f5c5540d124e7f8ddf607a3e5601be [term 1 FOLLOWER]: Refusing update from remote peer 97408e06e59a495894e71b51d054ec38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:31.805154 17845 raft_consensus.cc:1261] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 1 FOLLOWER]: Refusing update from remote peer 97408e06e59a495894e71b51d054ec38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:31.805886 17919 consensus_queue.cc:1035] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f5c5540d124e7f8ddf607a3e5601be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:31.806061 17915 consensus_queue.cc:1035] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a3205511f01452f93c78874d25da9d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:31.811889 17922 mvcc.cc:205] Tried to move back new op lower bound from 7298900835546136576 to 7298900835320164352. Current Snapshot: MvccSnapshot[applied={T|T < 7298900835546136576}]
I20260620 12:23:31.813575 17924 mvcc.cc:205] Tried to move back new op lower bound from 7298900835546136576 to 7298900835320164352. Current Snapshot: MvccSnapshot[applied={T|T < 7298900835546136576}]
I20260620 12:23:31.820722 17431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:31.820928 17431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:31.823184 17431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dcaa9ecff4244b2baf45f56bb0570b5: Sending DeleteTablet for 3 replicas of tablet 20da48f2eb964385a56b281987ddef33
I20260620 12:23:31.823995 17825 tablet_service.cc:1514] Processing DeleteTablet for tablet 20da48f2eb964385a56b281987ddef33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41814
I20260620 12:23:31.824008 17557 tablet_service.cc:1514] Processing DeleteTablet for tablet 20da48f2eb964385a56b281987ddef33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42992
I20260620 12:23:31.824523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 17494
I20260620 12:23:31.824635 17691 tablet_service.cc:1514] Processing DeleteTablet for tablet 20da48f2eb964385a56b281987ddef33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33110
I20260620 12:23:31.826071 17933 tablet_replica.cc:330] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: stopping tablet replica
I20260620 12:23:31.826432 17933 raft_consensus.cc:2229] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:23:31.826864 17933 raft_consensus.cc:2258] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:31.826982 17931 tablet_replica.cc:330] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: stopping tablet replica
I20260620 12:23:31.827229 17931 raft_consensus.cc:2229] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:31.827512 17931 raft_consensus.cc:2258] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:31.828719 17933 ts_tablet_manager.cc:1905] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:31.828876 17931 ts_tablet_manager.cc:1905] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:31.831564 17933 ts_tablet_manager.cc:1918] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:31.831694 17933 log.cc:1196] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/20da48f2eb964385a56b281987ddef33
I20260620 12:23:31.832044 17933 ts_tablet_manager.cc:1939] T 20da48f2eb964385a56b281987ddef33 P 97408e06e59a495894e71b51d054ec38: Deleting consensus metadata
I20260620 12:23:31.832402 17931 ts_tablet_manager.cc:1918] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:31.832561 17931 log.cc:1196] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/20da48f2eb964385a56b281987ddef33
I20260620 12:23:31.832950 17931 ts_tablet_manager.cc:1939] T 20da48f2eb964385a56b281987ddef33 P 7a3205511f01452f93c78874d25da9d9: Deleting consensus metadata
I20260620 12:23:31.833289 17418 catalog_manager.cc:4841] TS 97408e06e59a495894e71b51d054ec38 (127.0.98.2:33961): tablet 20da48f2eb964385a56b281987ddef33 (table test-table [id=7b81189b3841436f827438e45baf410d]) successfully deleted
I20260620 12:23:31.833539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 17628
W20260620 12:23:31.833599 17419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43995 (error 108)
I20260620 12:23:31.834988 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:31.834696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43995 (local address 127.0.0.1:43000)
0620 12:23:31.834824 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:31.834905 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:23:31.835537 17419 catalog_manager.cc:4584] TS 90f5c5540d124e7f8ddf607a3e5601be (127.0.98.1:43995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20da48f2eb964385a56b281987ddef33: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43995: connect: Connection refused (error 111)
I20260620 12:23:31.835741 17418 catalog_manager.cc:4841] TS 7a3205511f01452f93c78874d25da9d9 (127.0.98.3:38553): tablet 20da48f2eb964385a56b281987ddef33 (table test-table [id=7b81189b3841436f827438e45baf410d]) successfully deleted
I20260620 12:23:31.839146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 17762
I20260620 12:23:31.844491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 17400
2026-06-20T12:23:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2511 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T12:23:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:31Z Disabled control of system clock
I20260620 12:23:31.874099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43685
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:23:31.982322 17938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:31.982585 17938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:31.982620 17938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:31.986629 17938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:31.986691 17938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:31.986707 17938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:31.986725 17938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:31.986742 17938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:31.991540 17938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:31.992658 17938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:31.994050 17938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:32.002264 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:32.002315 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:32.002383 17938 server_base.cc:1034] running on GCE node
W20260620 12:23:32.002300 17946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:32.002920 17938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:32.003909 17938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:32.005146 17938 hybrid_clock.cc:648] HybridClock initialized: now 1781958212005130 us; error 26 us; skew 500 ppm
I20260620 12:23:32.006922 17938 webserver.cc:460] Webserver started at http://127.0.98.62:33347/ using document root <none> and password file <none>
I20260620 12:23:32.007478 17938 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:32.007567 17938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:32.007791 17938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:32.009883 17938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "3b6db40e5c4f4e52a89fcaafb5812614"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "2ffe2912d5cd18bccce0bbb2396f945f"
server_key_iv: "b7f4ec19a960c09c7952346175866f4e"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.010429 17938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b6db40e5c4f4e52a89fcaafb5812614"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "2ffe2912d5cd18bccce0bbb2396f945f"
server_key_iv: "b7f4ec19a960c09c7952346175866f4e"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.013967 17938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:32.016311 17952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.017535 17938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:32.017669 17938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "3b6db40e5c4f4e52a89fcaafb5812614"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "2ffe2912d5cd18bccce0bbb2396f945f"
server_key_iv: "b7f4ec19a960c09c7952346175866f4e"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.017774 17938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:32.030141 17938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:32.030736 17938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:32.030918 17938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:32.037636 17938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43685
I20260620 12:23:32.037680 18004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43685 every 8 connection(s)
I20260620 12:23:32.038878 17938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:32.039348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 17938
I20260620 12:23:32.039516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:32.039880 392 external_mini_cluster.cc:1428] Setting key 05d40338ffe73296e6ca91981345be75
I20260620 12:23:32.044107 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:32.050683 18005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614: Bootstrap starting.
I20260620 12:23:32.051345 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.042196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.0.98.62:43685)
0620 12:23:32.042815 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.042830 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.042871 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:23:32.043774 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.043799 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.044329 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.044713 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:32.045922 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.046884 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.048114 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.048553 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.048665 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.049563 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:32.049568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:32.049579 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:32.049965 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:32.050017 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:32.050036 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:32.050111 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.050282 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.050582 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":152,"threads_started":1}
I20260620 12:23:32.052986 18005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:32.053869 18005 log.cc:824] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:32.055612 18005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614: No bootstrap required, opened a new log
I20260620 12:23:32.059194 18005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6db40e5c4f4e52a89fcaafb5812614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43685 } }
I20260620 12:23:32.059392 18005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:32.059425 18005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6db40e5c4f4e52a89fcaafb5812614, State: Initialized, Role: FOLLOWER
I20260620 12:23:32.059748 18005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6db40e5c4f4e52a89fcaafb5812614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43685 } }
I20260620 12:23:32.059857 18005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:32.059895 18005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:32.059964 18005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:32.060721 18005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6db40e5c4f4e52a89fcaafb5812614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43685 } }
I20260620 12:23:32.061000 18005 leader_election.cc:302] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b6db40e5c4f4e52a89fcaafb5812614; no voters:
I20260620 12:23:32.061302 18005 leader_election.cc:288] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:32.061445 18010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:32.061669 18010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [term 1 LEADER]: Becoming Leader. State: Replica: 3b6db40e5c4f4e52a89fcaafb5812614, State: Running, Role: LEADER
I20260620 12:23:32.061990 18010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6db40e5c4f4e52a89fcaafb5812614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43685 } }
I20260620 12:23:32.062438 18005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:32.063498 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b6db40e5c4f4e52a89fcaafb5812614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6db40e5c4f4e52a89fcaafb5812614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43685 } } }
I20260620 12:23:32.063618 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:32.063946 18012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b6db40e5c4f4e52a89fcaafb5812614. Latest consensus state: current_term: 1 leader_uuid: "3b6db40e5c4f4e52a89fcaafb5812614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6db40e5c4f4e52a89fcaafb5812614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43685 } } }
I20260620 12:23:32.064092 18012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:32.064390 18019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:32.067018 18019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:32.072103 18019 catalog_manager.cc:1347] Generated new cluster ID: 8e7cf3a06c064508a115a62eb6fa3769
I20260620 12:23:32.072165 18019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:32.087402 18019 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:32.088568 18019 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:32.095314 18019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614: Generated new TSK 0
I20260620 12:23:32.096052 18019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:32.105264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43685
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:23:32.215914 18029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:32.216172 18029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:32.216241 18029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:32.220129 18029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:32.220201 18029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:32.220283 18029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:23:32.225100 18029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:32.226096 18029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:32.227404 18029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:32.236542 18034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:32.236577 18035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:32.236598 18037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:32.236862 18029 server_base.cc:1034] running on GCE node
I20260620 12:23:32.237280 18029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:32.237922 18029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:32.239102 18029 hybrid_clock.cc:648] HybridClock initialized: now 1781958212239075 us; error 49 us; skew 500 ppm
I20260620 12:23:32.240857 18029 webserver.cc:460] Webserver started at http://127.0.98.1:40041/ using document root <none> and password file <none>
I20260620 12:23:32.241478 18029 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:32.241569 18029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:32.241789 18029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:32.243745 18029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "3bfb0437461048db93df2c866ba7ca7d"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "b2d0e27c0d6ced27f36987ec89d88eaa"
server_key_iv: "ed8dce411bc48c79b2d61ac6e184a86d"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.244302 18029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3bfb0437461048db93df2c866ba7ca7d"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "b2d0e27c0d6ced27f36987ec89d88eaa"
server_key_iv: "ed8dce411bc48c79b2d61ac6e184a86d"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.247851 18029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:23:32.250280 18043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.251410 18029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:23:32.251538 18029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "3bfb0437461048db93df2c866ba7ca7d"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "b2d0e27c0d6ced27f36987ec89d88eaa"
server_key_iv: "ed8dce411bc48c79b2d61ac6e184a86d"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.251649 18029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:32.266139 18029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:32.266757 18029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:32.266954 18029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:32.267496 18029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:32.268388 18029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:32.268464 18029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.268534 18029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:32.268579 18029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.276773 18029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37395
I20260620 12:23:32.276794 18156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37395 every 8 connection(s)
I20260620 12:23:32.277621 18029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:23:32.282778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 18029
I20260620 12:23:32.282938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:23:32.283231 392 external_mini_cluster.cc:1428] Setting key 98fac8562746c70dd943adc6a3f2a480
I20260620 12:23:32.285236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43685
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:23:32.286844 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.279341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54163 (local address 127.0.98.62:43685)
0620 12:23:32.279480 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.279484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.280392 (+ 908us) server_negotiation.cc:407] Connection header received
0620 12:23:32.281761 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.281764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.281815 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.281887 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:32.282820 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.283517 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.284853 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.285105 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.285219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.286365 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:32.286368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:32.286370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:32.286380 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:32.286399 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:32.286408 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:32.286485 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.286639 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.286705 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:23:32.287599 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.279663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43685 (local address 127.0.98.1:54163)
0620 12:23:32.280264 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.280301 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:32.281585 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:32.282038 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:32.282045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:32.282123 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:32.282663 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.282682 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.283671 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.283674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.284729 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.284738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.285253 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.285257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.285493 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.286104 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:32.286106 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:32.286143 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:32.286200 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:32.286487 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:32.286499 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:32.286507 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:23:32.286638 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 12:23:32.286894 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
I20260620 12:23:32.288946 18157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43685
I20260620 12:23:32.289191 18157 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:32.289676 18157 heartbeater.cc:511] Master 127.0.98.62:43685 requested a full tablet report, sending...
I20260620 12:23:32.291128 17969 ts_manager.cc:194] Registered new tserver with Master: 3bfb0437461048db93df2c866ba7ca7d (127.0.98.1:37395)
I20260620 12:23:32.292245 17969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54163
W20260620 12:23:32.392280 18160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:32.392565 18160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:32.392632 18160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:32.396834 18160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:32.396914 18160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:32.397010 18160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:23:32.401866 18160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:32.402868 18160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:32.404206 18160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:32.413908 18165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:32.413910 18168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:32.414520 18160 server_base.cc:1034] running on GCE node
W20260620 12:23:32.414958 18166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:32.415367 18160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:32.416028 18160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:32.417194 18160 hybrid_clock.cc:648] HybridClock initialized: now 1781958212417186 us; error 31 us; skew 500 ppm
I20260620 12:23:32.418922 18160 webserver.cc:460] Webserver started at http://127.0.98.2:39979/ using document root <none> and password file <none>
I20260620 12:23:32.419497 18160 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:32.419588 18160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:32.419811 18160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:32.421947 18160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "37dca85c3dc4460a8d56c3f19e07c97e"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "8640001b7b2134cf59ba30a590e4178d"
server_key_iv: "a464b839dfc4989b30e4590a5e4e550d"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.422542 18160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "37dca85c3dc4460a8d56c3f19e07c97e"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "8640001b7b2134cf59ba30a590e4178d"
server_key_iv: "a464b839dfc4989b30e4590a5e4e550d"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.425999 18160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:23:32.428246 18174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.429534 18160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:23:32.429669 18160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "37dca85c3dc4460a8d56c3f19e07c97e"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "8640001b7b2134cf59ba30a590e4178d"
server_key_iv: "a464b839dfc4989b30e4590a5e4e550d"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.429778 18160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:32.447544 18160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:32.448251 18160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:32.448457 18160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:32.449095 18160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:32.450027 18160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:32.450098 18160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.450170 18160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:32.450212 18160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.459199 18160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42315
I20260620 12:23:32.459223 18287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42315 every 8 connection(s)
I20260620 12:23:32.460139 18160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:23:32.461294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 18160
I20260620 12:23:32.461416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:23:32.461694 392 external_mini_cluster.cc:1428] Setting key ac6a2a31510b1ee573901a8fbace3da7
I20260620 12:23:32.463728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43685
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:23:32.471108 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.462238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41973 (local address 127.0.98.62:43685)
0620 12:23:32.462415 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.462421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.463698 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 12:23:32.465406 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.465408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.465453 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.465511 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:32.466561 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.467075 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.468948 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.469214 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.469311 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.470642 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:32.470644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:32.470646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:32.470658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:32.470676 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:32.470683 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:32.470741 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.470937 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.470992 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 12:23:32.472389 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.462737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43685 (local address 127.0.98.2:41973)
0620 12:23:32.463527 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.463579 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:32.465184 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:32.465702 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:32.465712 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:32.465823 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:32.466440 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.466453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.467228 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.467234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.468812 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.468827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.469394 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.469402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.469642 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.470305 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:32.470309 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:32.470380 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:32.470473 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:32.470771 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:32.470783 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:32.470790 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 12:23:32.470934 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 12:23:32.471257 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":171,"threads_started":1}
I20260620 12:23:32.473621 18288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43685
I20260620 12:23:32.473855 18288 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:32.474344 18288 heartbeater.cc:511] Master 127.0.98.62:43685 requested a full tablet report, sending...
I20260620 12:23:32.475349 17969 ts_manager.cc:194] Registered new tserver with Master: 37dca85c3dc4460a8d56c3f19e07c97e (127.0.98.2:42315)
I20260620 12:23:32.475920 17969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41973
W20260620 12:23:32.573706 18291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:32.573951 18291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:32.573987 18291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:32.578555 18291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:32.578605 18291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:32.578689 18291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:23:32.583343 18291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:32.584337 18291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:32.585759 18291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:32.595323 18296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:32.595337 18299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:32.595357 18291 server_base.cc:1034] running on GCE node
W20260620 12:23:32.595324 18297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:32.595944 18291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:32.596571 18291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:32.597721 18291 hybrid_clock.cc:648] HybridClock initialized: now 1781958212597701 us; error 38 us; skew 500 ppm
I20260620 12:23:32.599395 18291 webserver.cc:460] Webserver started at http://127.0.98.3:36223/ using document root <none> and password file <none>
I20260620 12:23:32.599915 18291 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:32.599977 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:32.600129 18291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:32.602079 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "140c59a9f2184b77ac98b3c612e30a8f"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "a937575b69ad7b722e43e61893346c2a"
server_key_iv: "da58219d3fc55bea56278d188b78c51f"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.602566 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "140c59a9f2184b77ac98b3c612e30a8f"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "a937575b69ad7b722e43e61893346c2a"
server_key_iv: "da58219d3fc55bea56278d188b78c51f"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.605942 18291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:32.608125 18305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.609334 18291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:32.609488 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "140c59a9f2184b77ac98b3c612e30a8f"
format_stamp: "Formatted at 2026-06-20 12:23:32 on dist-test-slave-9b3m"
server_key: "a937575b69ad7b722e43e61893346c2a"
server_key_iv: "da58219d3fc55bea56278d188b78c51f"
server_key_version: "encryptionkey@0"
I20260620 12:23:32.609597 18291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:32.642793 18291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:32.643522 18291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:32.643725 18291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:32.644330 18291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:32.645344 18291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:32.645413 18291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.645532 18291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:32.645589 18291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:32.654567 18291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37817
I20260620 12:23:32.654595 18418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37817 every 8 connection(s)
I20260620 12:23:32.655514 18291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:23:32.660764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 18291
I20260620 12:23:32.660878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:23:32.661222 392 external_mini_cluster.cc:1428] Setting key 831d7d71438751580469cc32b91e4600
I20260620 12:23:32.665199 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.657290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46055 (local address 127.0.98.62:43685)
0620 12:23:32.657471 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.657476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.658466 (+ 990us) server_negotiation.cc:407] Connection header received
0620 12:23:32.659757 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.659760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.659806 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.659879 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:32.660761 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.661564 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.662999 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.663330 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.663463 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.664622 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:32.664626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:32.664628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:32.664641 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:32.664681 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:32.664693 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:32.664754 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.664977 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.665034 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 12:23:32.665997 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.657704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43685 (local address 127.0.98.3:46055)
0620 12:23:32.658315 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.658354 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:32.659573 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:32.660052 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:32.660059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:32.660139 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:32.660616 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.660629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.661696 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.661699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.662871 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.662881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.663481 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.663485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.663678 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.664319 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:32.664321 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:32.664361 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:32.664423 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:32.664767 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:32.664776 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:32.664782 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 12:23:32.664964 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 12:23:32.665241 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260620 12:23:32.667073 18419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43685
I20260620 12:23:32.667284 18419 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:32.667766 18419 heartbeater.cc:511] Master 127.0.98.62:43685 requested a full tablet report, sending...
I20260620 12:23:32.668757 17969 ts_manager.cc:194] Registered new tserver with Master: 140c59a9f2184b77ac98b3c612e30a8f (127.0.98.3:37817)
I20260620 12:23:32.669395 17969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46055
I20260620 12:23:32.674858 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:23:32.680182 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.676683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60884 (local address 127.0.98.62:43685)
0620 12:23:32.676828 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.676833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.676979 (+ 146us) server_negotiation.cc:407] Connection header received
0620 12:23:32.677119 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.677121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.677160 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.677224 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:32.677465 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.677955 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.678730 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.678951 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.679025 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.679776 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:32.679779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:32.679781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:32.679792 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:32.679810 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:32.679818 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:32.679896 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.679958 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.680000 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 12:23:32.682760 17969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:23:32.685773 17969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:23:32.700508 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.694726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37817 (local address 127.0.0.1:51438)
0620 12:23:32.695641 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.695655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:32.695750 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:32.696348 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:32.696356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:32.696383 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:32.696522 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.696532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.698065 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.698068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.699070 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.699080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.699967 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.699971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.700073 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.700112 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 12:23:32.700227 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 12:23:32.700340 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260620 12:23:32.700562 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.695220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.0.98.3:37817)
0620 12:23:32.695852 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.695863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.695887 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:23:32.695970 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.695977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.696186 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.696341 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:32.696658 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.697921 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.699198 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.699838 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.699941 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.700117 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.700270 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.700367 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":140,"threads_started":1}
I20260620 12:23:32.701145 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.694433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37395 (local address 127.0.0.1:42078)
0620 12:23:32.695352 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.695403 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:32.695542 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:32.696790 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:32.696795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:32.696814 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:32.696879 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.696883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.697907 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.697913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.699485 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.699496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.700723 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.700729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.700823 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.700845 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:23:32.700935 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:23:32.701001 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":112,"thread_start_us":101,"threads_started":1}
I20260620 12:23:32.701279 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.694614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.98.1:37395)
0620 12:23:32.696260 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.696273 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.696295 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:23:32.696364 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.696368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.696530 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.696664 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:32.696992 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.697757 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.699619 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.700382 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.700489 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.700659 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.700976 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.701092 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1543,"thread_start_us":117,"threads_started":1}
I20260620 12:23:32.701601 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.694770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42315 (local address 127.0.0.1:46570)
0620 12:23:32.695910 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.695933 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:32.696020 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:32.696518 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:32.696521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:32.696538 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:32.696660 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.696667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.698105 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.698108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.699485 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.699496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.700962 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.700966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.701075 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.701098 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:23:32.701162 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 12:23:32.701483 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":95,"thread_start_us":87,"threads_started":1}
I20260620 12:23:32.702205 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.695220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.0.98.2:42315)
0620 12:23:32.695658 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.695672 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.696027 (+ 355us) server_negotiation.cc:407] Connection header received
0620 12:23:32.696106 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.696110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.696248 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.696427 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:32.696773 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.697999 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.699636 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.700319 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.700410 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.701906 (+ 1496us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.701963 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.702057 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":105,"threads_started":1}
I20260620 12:23:32.703475 18353 tablet_service.cc:1467] Processing CreateTablet for tablet 18af314083e749c9b72c8f4560dabf39 (DEFAULT_TABLE table=test-table [id=4f83594e95384dceb797a4613120a40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:32.703684 18091 tablet_service.cc:1467] Processing CreateTablet for tablet 18af314083e749c9b72c8f4560dabf39 (DEFAULT_TABLE table=test-table [id=4f83594e95384dceb797a4613120a40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:32.704528 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18af314083e749c9b72c8f4560dabf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:32.704552 18353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18af314083e749c9b72c8f4560dabf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:32.704644 18222 tablet_service.cc:1467] Processing CreateTablet for tablet 18af314083e749c9b72c8f4560dabf39 (DEFAULT_TABLE table=test-table [id=4f83594e95384dceb797a4613120a40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:32.705571 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18af314083e749c9b72c8f4560dabf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:32.711024 18436 tablet_bootstrap.cc:492] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d: Bootstrap starting.
I20260620 12:23:32.712221 18437 tablet_bootstrap.cc:492] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: Bootstrap starting.
I20260620 12:23:32.713212 18436 tablet_bootstrap.cc:654] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:32.713356 18438 tablet_bootstrap.cc:492] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e: Bootstrap starting.
I20260620 12:23:32.713994 18436 log.cc:824] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:32.714287 18437 tablet_bootstrap.cc:654] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:32.715061 18437 log.cc:824] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:32.715369 18438 tablet_bootstrap.cc:654] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:32.715978 18438 log.cc:824] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:32.716398 18436 tablet_bootstrap.cc:492] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d: No bootstrap required, opened a new log
I20260620 12:23:32.716594 18436 ts_tablet_manager.cc:1397] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 12:23:32.717978 18437 tablet_bootstrap.cc:492] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: No bootstrap required, opened a new log
I20260620 12:23:32.718170 18437 ts_tablet_manager.cc:1397] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 12:23:32.718412 18438 tablet_bootstrap.cc:492] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e: No bootstrap required, opened a new log
I20260620 12:23:32.718592 18438 ts_tablet_manager.cc:1397] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:23:32.720614 18436 raft_consensus.cc:348] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.720868 18436 raft_consensus.cc:374] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:32.720922 18436 raft_consensus.cc:729] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfb0437461048db93df2c866ba7ca7d, State: Initialized, Role: FOLLOWER
I20260620 12:23:32.721392 18436 consensus_queue.cc:260] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.721433 18437 raft_consensus.cc:348] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.721668 18437 raft_consensus.cc:374] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:32.721725 18437 raft_consensus.cc:729] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140c59a9f2184b77ac98b3c612e30a8f, State: Initialized, Role: FOLLOWER
I20260620 12:23:32.722167 18437 consensus_queue.cc:260] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.722239 18157 heartbeater.cc:503] Master 127.0.98.62:43685 was elected leader, sending a full tablet report...
I20260620 12:23:32.722409 18436 ts_tablet_manager.cc:1428] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 12:23:32.723030 18437 ts_tablet_manager.cc:1428] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:23:32.723062 18438 raft_consensus.cc:348] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.723338 18438 raft_consensus.cc:374] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:32.723372 18419 heartbeater.cc:503] Master 127.0.98.62:43685 was elected leader, sending a full tablet report...
I20260620 12:23:32.723392 18438 raft_consensus.cc:729] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dca85c3dc4460a8d56c3f19e07c97e, State: Initialized, Role: FOLLOWER
I20260620 12:23:32.723775 18438 consensus_queue.cc:260] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.724431 18288 heartbeater.cc:503] Master 127.0.98.62:43685 was elected leader, sending a full tablet report...
I20260620 12:23:32.724669 18438 ts_tablet_manager.cc:1428] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 12:23:32.778748 18158 tablet.cc:2369] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:32.857671 18443 raft_consensus.cc:482] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:32.857908 18443 raft_consensus.cc:504] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.859055 18443 leader_election.cc:288] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bfb0437461048db93df2c866ba7ca7d (127.0.98.1:37395), 37dca85c3dc4460a8d56c3f19e07c97e (127.0.98.2:42315)
I20260620 12:23:32.862740 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.859422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33621 (local address 127.0.98.2:42315)
0620 12:23:32.859564 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.859570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.859589 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:23:32.859721 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.859724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.859777 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.859864 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:32.860143 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.860643 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.861747 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.862205 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.862321 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.862379 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.862559 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.862621 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:23:32.862741 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.859320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42315 (local address 127.0.98.3:33621)
0620 12:23:32.859448 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.859465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:32.859590 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:32.859876 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:32.859879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:32.859910 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:32.860031 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.860037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.860756 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.860760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.861624 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.861633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.862325 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.862329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.862433 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.862451 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:23:32.862543 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 12:23:32.862612 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"spinlock_wait_cycles":11648}
I20260620 12:23:32.863080 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.859320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37395 (local address 127.0.98.3:53977)
0620 12:23:32.859773 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.859784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:32.859926 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:32.860221 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:32.860223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:32.860238 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:32.860370 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.860375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.861096 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.861100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.861858 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:32.861866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.862623 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.862627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:32.862722 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.862747 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:23:32.862834 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:23:32.862921 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":129,"spinlock_wait_cycles":5248,"thread_start_us":105,"threads_started":1}
I20260620 12:23:32.863189 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.859424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53977 (local address 127.0.98.1:37395)
0620 12:23:32.859552 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.859556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.859881 (+ 325us) server_negotiation.cc:407] Connection header received
0620 12:23:32.860080 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.860083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.860127 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.860212 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:32.860496 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.860970 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.861988 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.862431 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.862617 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.862673 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.862998 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.863068 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:23:32.863603 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18af314083e749c9b72c8f4560dabf39" candidate_uuid: "140c59a9f2184b77ac98b3c612e30a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" is_pre_election: true
I20260620 12:23:32.863735 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18af314083e749c9b72c8f4560dabf39" candidate_uuid: "140c59a9f2184b77ac98b3c612e30a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfb0437461048db93df2c866ba7ca7d" is_pre_election: true
I20260620 12:23:32.863935 18242 raft_consensus.cc:2454] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140c59a9f2184b77ac98b3c612e30a8f in term 0.
I20260620 12:23:32.864074 18111 raft_consensus.cc:2454] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140c59a9f2184b77ac98b3c612e30a8f in term 0.
I20260620 12:23:32.864420 18309 leader_election.cc:302] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140c59a9f2184b77ac98b3c612e30a8f, 37dca85c3dc4460a8d56c3f19e07c97e; no voters:
I20260620 12:23:32.864703 18443 raft_consensus.cc:2781] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:23:32.864781 18443 raft_consensus.cc:482] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:23:32.864812 18443 raft_consensus.cc:3037] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:32.865831 18443 raft_consensus.cc:504] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.866204 18443 leader_election.cc:288] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [CANDIDATE]: Term 1 election: Requested vote from peers 3bfb0437461048db93df2c866ba7ca7d (127.0.98.1:37395), 37dca85c3dc4460a8d56c3f19e07c97e (127.0.98.2:42315)
I20260620 12:23:32.866530 18242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18af314083e749c9b72c8f4560dabf39" candidate_uuid: "140c59a9f2184b77ac98b3c612e30a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dca85c3dc4460a8d56c3f19e07c97e"
I20260620 12:23:32.866552 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18af314083e749c9b72c8f4560dabf39" candidate_uuid: "140c59a9f2184b77ac98b3c612e30a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfb0437461048db93df2c866ba7ca7d"
I20260620 12:23:32.866631 18242 raft_consensus.cc:3037] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:32.866665 18111 raft_consensus.cc:3037] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:32.867600 18111 raft_consensus.cc:2454] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140c59a9f2184b77ac98b3c612e30a8f in term 1.
I20260620 12:23:32.867663 18242 raft_consensus.cc:2454] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140c59a9f2184b77ac98b3c612e30a8f in term 1.
I20260620 12:23:32.867911 18306 leader_election.cc:302] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140c59a9f2184b77ac98b3c612e30a8f, 3bfb0437461048db93df2c866ba7ca7d; no voters:
I20260620 12:23:32.868069 18443 raft_consensus.cc:2781] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:32.868250 18443 raft_consensus.cc:686] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 1 LEADER]: Becoming Leader. State: Replica: 140c59a9f2184b77ac98b3c612e30a8f, State: Running, Role: LEADER
I20260620 12:23:32.868548 18443 consensus_queue.cc:237] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } }
I20260620 12:23:32.871676 17968 catalog_manager.cc:5495] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 140c59a9f2184b77ac98b3c612e30a8f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "140c59a9f2184b77ac98b3c612e30a8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "140c59a9f2184b77ac98b3c612e30a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 } health_report { overall_health: UNKNOWN } } }
I20260620 12:23:32.887529 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.883877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.0.98.3:37817)
0620 12:23:32.884004 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:32.884011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:32.884049 (+ 38us) server_negotiation.cc:407] Connection header received
0620 12:23:32.884254 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:32.884256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:32.884307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:32.884431 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:23:32.884663 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.885234 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.886092 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:32.886313 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:32.886433 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:32.886733 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:23:32.887125 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:23:32.887254 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:32.887329 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 12:23:32.887387 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:23:32.893374 18242 raft_consensus.cc:1261] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 1 FOLLOWER]: Refusing update from remote peer 140c59a9f2184b77ac98b3c612e30a8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:32.893435 18111 raft_consensus.cc:1261] T 18af314083e749c9b72c8f4560dabf39 P 3bfb0437461048db93df2c866ba7ca7d [term 1 FOLLOWER]: Refusing update from remote peer 140c59a9f2184b77ac98b3c612e30a8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:32.894176 18446 consensus_queue.cc:1035] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfb0437461048db93df2c866ba7ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:32.894351 18443 consensus_queue.cc:1035] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dca85c3dc4460a8d56c3f19e07c97e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:32.901733 18449 mvcc.cc:205] Tried to move back new op lower bound from 7298900840003129344 to 7298900839912112128. Current Snapshot: MvccSnapshot[applied={T|T < 7298900840003129344}]
I20260620 12:23:32.903908 18450 mvcc.cc:205] Tried to move back new op lower bound from 7298900840003129344 to 7298900839912112128. Current Snapshot: MvccSnapshot[applied={T|T < 7298900840003129344 or (T in {7298900840003129344})}]
I20260620 12:23:32.911789 17967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:32.911954 17967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:32.914342 17967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b6db40e5c4f4e52a89fcaafb5812614: Sending DeleteTablet for 3 replicas of tablet 18af314083e749c9b72c8f4560dabf39
I20260620 12:23:32.915030 18091 tablet_service.cc:1514] Processing DeleteTablet for tablet 18af314083e749c9b72c8f4560dabf39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:32 UTC) from {username='slave'} at 127.0.0.1:42078
I20260620 12:23:32.915226 18353 tablet_service.cc:1514] Processing DeleteTablet for tablet 18af314083e749c9b72c8f4560dabf39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:32 UTC) from {username='slave'} at 127.0.0.1:51438
I20260620 12:23:32.915257 18222 tablet_service.cc:1514] Processing DeleteTablet for tablet 18af314083e749c9b72c8f4560dabf39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:32 UTC) from {username='slave'} at 127.0.0.1:46570
I20260620 12:23:32.915658 18459 tablet_replica.cc:330] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: stopping tablet replica
I20260620 12:23:32.915953 18459 raft_consensus.cc:2229] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:23:32.916288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 18029
I20260620 12:23:32.916337 18459 raft_consensus.cc:2258] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:32.919169 18459 ts_tablet_manager.cc:1905] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:32.921181 18460 tablet_replica.cc:330] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e: stopping tablet replica
I20260620 12:23:32.921566 18460 raft_consensus.cc:2229] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:32.921934 18460 raft_consensus.cc:2258] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:32.922758 18459 ts_tablet_manager.cc:1918] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:32.922915 18459 log.cc:1196] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/18af314083e749c9b72c8f4560dabf39
I20260620 12:23:32.923331 18459 ts_tablet_manager.cc:1939] T 18af314083e749c9b72c8f4560dabf39 P 140c59a9f2184b77ac98b3c612e30a8f: Deleting consensus metadata
I20260620 12:23:32.923444 18460 ts_tablet_manager.cc:1905] T 18af314083e749c9b72c8f4560dabf39 P 37dca85c3dc4460a8d56c3f19e07c97e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 12:23:32.924048 17953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37395 (error 108)
I20260620 12:23:32.924811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 18160
I20260620 12:23:32.926641 17954 catalog_manager.cc:4841] TS 140c59a9f2184b77ac98b3c612e30a8f (127.0.98.3:37817): tablet 18af314083e749c9b72c8f4560dabf39 (table test-table [id=4f83594e95384dceb797a4613120a40a]) successfully deleted
I20260620 12:23:32.927212 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.926925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37395 (local address 127.0.0.1:42092)
0620 12:23:32.927055 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.927126 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 12:23:32.927400 17953 catalog_manager.cc:4584] TS 3bfb0437461048db93df2c866ba7ca7d (127.0.98.1:37395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18af314083e749c9b72c8f4560dabf39: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37395: connect: Connection refused (error 111)
I20260620 12:23:32.932441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 18291
I20260620 12:23:32.932683 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:32.932414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42315 (local address 127.0.0.1:46584)
0620 12:23:32.932548 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:32.932604 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 12:23:32.937017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 17938
2026-06-20T12:23:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1091 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:32 dist-test-slave-9b3m krb5kdc[18464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:32 dist-test-slave-9b3m krb5kdc[18464](info): set up 2 sockets
Jun 20 12:23:32 dist-test-slave-9b3m krb5kdc[18464](info): commencing operation
krb5kdc: starting...
W20260620 12:23:33.965188 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:34 dist-test-slave-9b3m krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:23:34.083649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40969
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:34.188917 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:34.189215 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:34.189291 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:34.193241 18477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:34.193318 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:34.193353 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:34.193389 18477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:34.193423 18477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:34.198572 18477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:34.199542 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:34.200480 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:34.208549 18482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:34.208566 18483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:34.208652 18477 server_base.cc:1034] running on GCE node
W20260620 12:23:34.208566 18485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:34.209421 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:34.210460 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:34.211678 18477 hybrid_clock.cc:648] HybridClock initialized: now 1781958214211648 us; error 39 us; skew 500 ppm
Jun 20 12:23:34 dist-test-slave-9b3m krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:34.217862 18477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:23:34.218693 18477 webserver.cc:460] Webserver started at http://127.0.98.62:37379/ using document root <none> and password file <none>
I20260620 12:23:34.219228 18477 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:34.219317 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:34.219539 18477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:34.221534 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "76c3485bef5b4f50836190aae242a2c6"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "7982e72ffba32d6a28473b65359fe3a7"
server_key_iv: "6527d8080847636c3225266852b1aa3f"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.222072 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "76c3485bef5b4f50836190aae242a2c6"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "7982e72ffba32d6a28473b65359fe3a7"
server_key_iv: "6527d8080847636c3225266852b1aa3f"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.225548 18477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:23:34.227787 18492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:34.228870 18477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:34.229004 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "76c3485bef5b4f50836190aae242a2c6"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "7982e72ffba32d6a28473b65359fe3a7"
server_key_iv: "6527d8080847636c3225266852b1aa3f"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.229179 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:34.262154 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:34.265409 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:34.265614 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:34.272507 18477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40969
I20260620 12:23:34.272532 18544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40969 every 8 connection(s)
I20260620 12:23:34.273438 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:34.275991 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:34.279382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 18477
I20260620 12:23:34.279515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:34.279781 392 external_mini_cluster.cc:1428] Setting key 53a8cd05d1890740026d114f1fb5c98d
I20260620 12:23:34.281554 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6: Bootstrap starting.
I20260620 12:23:34.283574 18545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:34.284277 18545 log.cc:824] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:34.286051 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6: No bootstrap required, opened a new log
Jun 20 12:23:34 dist-test-slave-9b3m krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:34.290692 18545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3485bef5b4f50836190aae242a2c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } }
I20260620 12:23:34.290957 18545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:34.291021 18545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c3485bef5b4f50836190aae242a2c6, State: Initialized, Role: FOLLOWER
I20260620 12:23:34.291493 18545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3485bef5b4f50836190aae242a2c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } }
I20260620 12:23:34.291658 18545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:34.291723 18545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:34.291819 18545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:34.292881 18545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3485bef5b4f50836190aae242a2c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } }
I20260620 12:23:34.293248 18545 leader_election.cc:302] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76c3485bef5b4f50836190aae242a2c6; no voters:
I20260620 12:23:34.293541 18545 leader_election.cc:288] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:34.293689 18550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:34.293901 18550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [term 1 LEADER]: Becoming Leader. State: Replica: 76c3485bef5b4f50836190aae242a2c6, State: Running, Role: LEADER
I20260620 12:23:34.294018 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:34.281147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.0.98.62:40969)
0620 12:23:34.281539 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:34.281553 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:34.281587 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:23:34.282237 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:34.282254 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:34.282563 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:34.282879 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:34.283654 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.284750 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.285660 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.286108 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.286273 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:34.288904 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:34.288911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:34.288930 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:34.288968 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:34.290868 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:34.291379 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:34.291384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:34.291389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:34.291521 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:34.291874 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:34.291878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:34.291880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:34.292358 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:34.292565 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:34.292774 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 12:23:34.293084 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260620 12:23:34.294236 18550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3485bef5b4f50836190aae242a2c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } }
I20260620 12:23:34.294718 18545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:34.295805 18552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76c3485bef5b4f50836190aae242a2c6. Latest consensus state: current_term: 1 leader_uuid: "76c3485bef5b4f50836190aae242a2c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3485bef5b4f50836190aae242a2c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } }
I20260620 12:23:34.295933 18552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:34.296236 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76c3485bef5b4f50836190aae242a2c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3485bef5b4f50836190aae242a2c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } }
I20260620 12:23:34.296311 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:34.296317 18559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:34.299381 18559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:34.309482 18559 catalog_manager.cc:1347] Generated new cluster ID: d8e08fd59b104cf6b280e989e8faf782
I20260620 12:23:34.309540 18559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:34.335719 18559 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:34.336993 18559 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:34.342252 18559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6: Generated new TSK 0
I20260620 12:23:34.342985 18559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:23:34.378389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:34.513964 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:34.514242 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:34.514322 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:34.518525 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:34.518608 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:34.518693 18571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:23:34.523945 18571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:34.524988 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:34.526001 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:34.535655 18576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:34.535660 18579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:34.535971 18571 server_base.cc:1034] running on GCE node
W20260620 12:23:34.535660 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:34.536484 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:34.537164 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:34.538316 18571 hybrid_clock.cc:648] HybridClock initialized: now 1781958214538287 us; error 42 us; skew 500 ppm
Jun 20 12:23:34 dist-test-slave-9b3m krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:34.544296 18571 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:23:34.545157 18571 webserver.cc:460] Webserver started at http://127.0.98.1:43103/ using document root <none> and password file <none>
I20260620 12:23:34.545666 18571 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:34.545722 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:34.545877 18571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:34.547775 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "b0dc571266c947f48d214991526d1570"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "4026d9523e70fa7483c3908a82b29baf"
server_key_iv: "86f8a0542f9d1893170fe1581fd0aa39"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.548282 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b0dc571266c947f48d214991526d1570"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "4026d9523e70fa7483c3908a82b29baf"
server_key_iv: "86f8a0542f9d1893170fe1581fd0aa39"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.551864 18571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:23:34.554320 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:34.555464 18571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:23:34.555594 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "b0dc571266c947f48d214991526d1570"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "4026d9523e70fa7483c3908a82b29baf"
server_key_iv: "86f8a0542f9d1893170fe1581fd0aa39"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.555699 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:34.583242 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:34.586303 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:34.586494 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:34.587059 18571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:34.587826 18571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:34.587872 18571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:34.587910 18571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:34.587934 18571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:34.596616 18571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36983
I20260620 12:23:34.596643 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36983 every 8 connection(s)
I20260620 12:23:34.597514 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:23:34.604972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 18571
I20260620 12:23:34.605134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:23:34.605480 392 external_mini_cluster.cc:1428] Setting key 6a0cf378145ad05ea9e9baa0a898b185
Jun 20 12:23:34 dist-test-slave-9b3m krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:34.611030 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:34.599283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52139 (local address 127.0.98.62:40969)
0620 12:23:34.599413 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:34.599417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:34.600197 (+ 780us) server_negotiation.cc:407] Connection header received
0620 12:23:34.601546 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:34.601550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:34.601607 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:34.601723 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:34.602814 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.603353 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.604473 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.604708 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.604819 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:34.607518 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:34.607522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:34.607525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:34.607555 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:34.609017 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:34.609562 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:34.609567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:34.609569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:34.609648 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:34.609992 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:34.609997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:34.609999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:34.610171 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:34.610279 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:34.610747 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 12:23:34.610866 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:23:34.611763 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:34.599642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.1:52139)
0620 12:23:34.600057 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:34.600098 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:34.601339 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:34.601876 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:34.601883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:34.602228 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:34.602671 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:34.602684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.603477 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.603487 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:34.604348 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:34.604357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.604856 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.604860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:34.605043 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:34.605722 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:34.605738 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:34.607363 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:34.609188 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:34.609194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:34.609215 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:34.609461 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:34.609759 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:34.609762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:34.609764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:34.609899 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:34.610278 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:34.610285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:34.610518 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 12:23:34.610698 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 12:23:34.610998 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260620 12:23:34.612884 18700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260620 12:23:34.613150 18700 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:34.613706 18700 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:23:34.615281 18509 ts_manager.cc:194] Registered new tserver with Master: b0dc571266c947f48d214991526d1570 (127.0.98.1:36983)
I20260620 12:23:34.616590 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:23:34.631439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:34.738583 18705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:34.738811 18705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:34.738852 18705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:34.742750 18705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:34.742798 18705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:34.742898 18705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:23:34.747985 18705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:34.748953 18705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:34.749943 18705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:34.759289 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:34.759333 18713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:34.759289 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:34.759577 18705 server_base.cc:1034] running on GCE node
I20260620 12:23:34.759940 18705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:34.760560 18705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:34.761734 18705 hybrid_clock.cc:648] HybridClock initialized: now 1781958214761719 us; error 27 us; skew 500 ppm
Jun 20 12:23:34 dist-test-slave-9b3m krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:34.764300 18705 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:23:34.765162 18705 webserver.cc:460] Webserver started at http://127.0.98.2:38595/ using document root <none> and password file <none>
I20260620 12:23:34.765702 18705 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:34.765787 18705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:34.766005 18705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:34.768011 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "eaa202c0c845432f9f6ea07bfb413ad6"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "138c0311e89ece3d9699355ad5fa1460"
server_key_iv: "5317c6ae31eb362e6bb2b7819de8f606"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.768559 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eaa202c0c845432f9f6ea07bfb413ad6"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "138c0311e89ece3d9699355ad5fa1460"
server_key_iv: "5317c6ae31eb362e6bb2b7819de8f606"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.772079 18705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:23:34.774403 18720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:34.775501 18705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:23:34.775626 18705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "eaa202c0c845432f9f6ea07bfb413ad6"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "138c0311e89ece3d9699355ad5fa1460"
server_key_iv: "5317c6ae31eb362e6bb2b7819de8f606"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.775741 18705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:34.785383 18705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:34.788064 18705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:34.788251 18705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:34.788789 18705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:34.789579 18705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:34.789649 18705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:34.789716 18705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:34.789759 18705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:34.798916 18705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45319
I20260620 12:23:34.798940 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45319 every 8 connection(s)
I20260620 12:23:34.799726 18705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:23:34.807104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 18705
I20260620 12:23:34.807228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:23:34.807510 392 external_mini_cluster.cc:1428] Setting key 39a6293bc2b4e417bcb31f70ffd03e4a
Jun 20 12:23:34 dist-test-slave-9b3m krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:34.813376 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:34.801446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53757 (local address 127.0.98.62:40969)
0620 12:23:34.801573 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:34.801577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:34.802349 (+ 772us) server_negotiation.cc:407] Connection header received
0620 12:23:34.803655 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:34.803659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:34.803716 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:34.803819 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:34.804896 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.805480 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.806597 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.806822 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.806947 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:34.809668 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:34.809675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:34.809681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:34.809715 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:34.811186 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:34.811808 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:34.811810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:34.811812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:34.811881 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:34.812249 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:34.812251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:34.812252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:34.812408 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:34.812480 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:34.813113 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 12:23:34.813247 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:23:34.814440 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:34.801776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.2:53757)
0620 12:23:34.802210 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:34.802249 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:34.803450 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:34.803980 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:34.803986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:34.804316 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:34.804753 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:34.804766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.805602 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.805607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:34.806464 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:34.806473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:34.807000 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:34.807003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:34.807141 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:34.807806 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:34.807822 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:34.809506 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:34.811310 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:34.811319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:34.811333 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:34.811703 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:34.811966 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:34.811971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:34.811976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:34.812153 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:34.812509 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:34.812518 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:34.812819 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 12:23:34.813030 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 12:23:34.813401 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260620 12:23:34.815586 18834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260620 12:23:34.815838 18834 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:34.816387 18834 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
I20260620 12:23:34.817510 18509 ts_manager.cc:194] Registered new tserver with Master: eaa202c0c845432f9f6ea07bfb413ad6 (127.0.98.2:45319)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 12:23:34.818189 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:23:34.835618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:34.947576 18839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:34.947844 18839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:34.947928 18839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:34.951921 18839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:34.952003 18839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:34.952095 18839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:23:34.957585 18839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:34.958590 18839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:34.959594 18839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:34.968711 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:34.968674 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:34.968659 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:34.968959 18839 server_base.cc:1034] running on GCE node
I20260620 12:23:34.969394 18839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:34.970057 18839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:34.971244 18839 hybrid_clock.cc:648] HybridClock initialized: now 1781958214971229 us; error 24 us; skew 500 ppm
Jun 20 12:23:34 dist-test-slave-9b3m krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:34.973991 18839 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:23:34.974829 18839 webserver.cc:460] Webserver started at http://127.0.98.3:35425/ using document root <none> and password file <none>
I20260620 12:23:34.975380 18839 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:34.975440 18839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:34.975594 18839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:34.977908 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5d17ef2c195404e93d163e83330b07f"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "88e32fac25208a7d1ac62669157395ac"
server_key_iv: "9bee3914b02b4ec4aa80ab671291699a"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.978441 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5d17ef2c195404e93d163e83330b07f"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "88e32fac25208a7d1ac62669157395ac"
server_key_iv: "9bee3914b02b4ec4aa80ab671291699a"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.981964 18839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:23:34.984233 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:34.985455 18839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:34.985567 18839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "f5d17ef2c195404e93d163e83330b07f"
format_stamp: "Formatted at 2026-06-20 12:23:34 on dist-test-slave-9b3m"
server_key: "88e32fac25208a7d1ac62669157395ac"
server_key_iv: "9bee3914b02b4ec4aa80ab671291699a"
server_key_version: "encryptionkey@0"
I20260620 12:23:34.985666 18839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:34.996809 18839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:34.999945 18839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:35.000144 18839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:35.000680 18839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:35.001549 18839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:35.001595 18839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:35.001662 18839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:35.001704 18839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:35.010185 18839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32981
I20260620 12:23:35.010205 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32981 every 8 connection(s)
I20260620 12:23:35.010983 18839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:23:35.011608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 18839
I20260620 12:23:35.011754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:23:35.012007 392 external_mini_cluster.cc:1428] Setting key a2c905860f0aa05730ec0c433f59bf86
Jun 20 12:23:35 dist-test-slave-9b3m krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:35.023460 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.012686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37211 (local address 127.0.98.62:40969)
0620 12:23:35.012823 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.012828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.013849 (+ 1021us) server_negotiation.cc:407] Connection header received
0620 12:23:35.015146 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.015149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.015193 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.015265 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:35.016303 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.016822 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.017908 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.018157 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.018242 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.020508 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:35.020511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:35.020513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:35.020536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:35.021871 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:35.022287 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:35.022289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:35.022290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:35.022357 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:35.022619 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:35.022621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:35.022622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:35.022788 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:35.022860 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.023252 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.023341 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 12:23:35.024214 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.013243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.98.3:37211)
0620 12:23:35.013678 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.013741 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:35.014938 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:35.015387 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:35.015393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:35.015729 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:35.016184 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.016196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.016918 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.016920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.017792 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.017800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.018298 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.018303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.018469 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.018950 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:35.018965 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:35.020356 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:35.021966 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:35.021970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:35.021980 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:35.022205 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:35.022427 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:35.022429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:35.022431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:35.022547 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:35.022867 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:35.022871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:35.023070 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 12:23:35.023210 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 12:23:35.023428 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":112,"threads_started":1}
I20260620 12:23:35.025388 18968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40969
I20260620 12:23:35.025615 18968 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:35.026158 18968 heartbeater.cc:511] Master 127.0.98.62:40969 requested a full tablet report, sending...
I20260620 12:23:35.027096 18508 ts_manager.cc:194] Registered new tserver with Master: f5d17ef2c195404e93d163e83330b07f (127.0.98.3:32981)
I20260620 12:23:35.027669 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37211
I20260620 12:23:35.037616 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:23:35.046367 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.039474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.0.98.62:40969)
0620 12:23:35.039568 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.039571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.039814 (+ 243us) server_negotiation.cc:407] Connection header received
0620 12:23:35.039893 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.039895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.039930 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.039993 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:35.040634 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.041150 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.041925 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.042121 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.042192 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.043618 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:35.043621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:35.043623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:35.043640 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:35.044862 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:35.045369 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:35.045371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:35.045373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:35.045440 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:35.045690 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:35.045692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:35.045693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:35.045874 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:35.045981 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.046186 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.046258 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 12:23:35.049098 18508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:23:35.052125 18508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:23:35.065855 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.059912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32981 (local address 127.0.0.1:47568)
0620 12:23:35.060653 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.060666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:35.060778 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:35.061500 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:35.061506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:35.061527 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:35.061656 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.061663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.062841 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.062844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.064216 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.064226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.065309 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.065313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.065424 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.065459 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 12:23:35.065569 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 12:23:35.065678 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":78,"thread_start_us":109,"threads_started":1}
I20260620 12:23:35.065872 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.060596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.98.3:32981)
0620 12:23:35.061054 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.061093 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.061116 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:23:35.061192 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.061195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.061355 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.061511 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:35.061786 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.062714 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.064350 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.065173 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.065274 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.065410 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.065612 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.065706 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":152,"threads_started":1}
I20260620 12:23:35.066190 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.059954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.0.98.1:36983)
0620 12:23:35.061077 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.061092 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.061116 (+ 24us) server_negotiation.cc:407] Connection header received
0620 12:23:35.061217 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.061223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.061436 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.061580 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:35.061887 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.063023 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.064659 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.065315 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.065414 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.065547 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.065924 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.066042 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":96,"threads_started":1}
I20260620 12:23:35.066466 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.060451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.0.98.2:45319)
0620 12:23:35.061076 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.061092 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.061922 (+ 830us) server_negotiation.cc:407] Connection header received
0620 12:23:35.062002 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.062007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.062173 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.062330 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:35.062526 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.063439 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.064717 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.065528 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.065647 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.065769 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.066199 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.066306 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":119,"threads_started":1}
I20260620 12:23:35.066129 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.059821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36983 (local address 127.0.0.1:55860)
0620 12:23:35.060605 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.060651 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:35.060776 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:35.061664 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:35.061667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:35.061682 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:35.061773 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.061785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.063155 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.063158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.064538 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.064547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.065604 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.065607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.065686 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.065704 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:23:35.065769 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 12:23:35.065823 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":27,"thread_start_us":77,"threads_started":1}
I20260620 12:23:35.067381 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.060190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:49558)
0620 12:23:35.061810 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.061824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:35.061908 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:35.062309 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:35.062312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:35.062330 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:35.062411 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.062416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.063561 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.063566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.064592 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.064623 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.065832 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.065836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.065944 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.065973 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 12:23:35.066040 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 12:23:35.066102 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"mutex_wait_us":5,"thread_start_us":78,"threads_started":1}
I20260620 12:23:35.069489 18902 tablet_service.cc:1467] Processing CreateTablet for tablet d4620dd48c5a4d1da5b09fb938d83be0 (DEFAULT_TABLE table=test-table [id=525382e0ac6b4edca75e0935809d66d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:35.069655 18634 tablet_service.cc:1467] Processing CreateTablet for tablet d4620dd48c5a4d1da5b09fb938d83be0 (DEFAULT_TABLE table=test-table [id=525382e0ac6b4edca75e0935809d66d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:35.070107 18768 tablet_service.cc:1467] Processing CreateTablet for tablet d4620dd48c5a4d1da5b09fb938d83be0 (DEFAULT_TABLE table=test-table [id=525382e0ac6b4edca75e0935809d66d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:35.070552 18902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4620dd48c5a4d1da5b09fb938d83be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:35.070559 18634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4620dd48c5a4d1da5b09fb938d83be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:35.070971 18768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4620dd48c5a4d1da5b09fb938d83be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:35.076666 18985 tablet_bootstrap.cc:492] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570: Bootstrap starting.
I20260620 12:23:35.078161 18986 tablet_bootstrap.cc:492] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Bootstrap starting.
I20260620 12:23:35.078912 18985 tablet_bootstrap.cc:654] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:35.079991 18985 log.cc:824] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:35.080047 18986 tablet_bootstrap.cc:654] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:35.080662 18987 tablet_bootstrap.cc:492] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Bootstrap starting.
I20260620 12:23:35.081174 18986 log.cc:824] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:35.082216 18985 tablet_bootstrap.cc:492] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570: No bootstrap required, opened a new log
I20260620 12:23:35.082402 18985 ts_tablet_manager.cc:1397] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:23:35.083042 18986 tablet_bootstrap.cc:492] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: No bootstrap required, opened a new log
I20260620 12:23:35.083215 18986 ts_tablet_manager.cc:1397] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 12:23:35.086314 18985 raft_consensus.cc:348] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.086582 18985 raft_consensus.cc:374] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:35.086675 18985 raft_consensus.cc:729] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0dc571266c947f48d214991526d1570, State: Initialized, Role: FOLLOWER
I20260620 12:23:35.086849 18987 tablet_bootstrap.cc:654] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:35.086915 18986 raft_consensus.cc:348] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.087167 18986 raft_consensus.cc:374] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:35.087134 18985 consensus_queue.cc:260] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.087263 18986 raft_consensus.cc:729] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d17ef2c195404e93d163e83330b07f, State: Initialized, Role: FOLLOWER
I20260620 12:23:35.087803 18700 heartbeater.cc:503] Master 127.0.98.62:40969 was elected leader, sending a full tablet report...
I20260620 12:23:35.088116 18985 ts_tablet_manager.cc:1428] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:23:35.088119 18987 log.cc:824] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:35.088300 18986 consensus_queue.cc:260] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.089267 18986 ts_tablet_manager.cc:1428] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 12:23:35.089692 18968 heartbeater.cc:503] Master 127.0.98.62:40969 was elected leader, sending a full tablet report...
I20260620 12:23:35.090308 18987 tablet_bootstrap.cc:492] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: No bootstrap required, opened a new log
I20260620 12:23:35.090487 18987 ts_tablet_manager.cc:1397] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 12:23:35.093539 18987 raft_consensus.cc:348] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.093755 18987 raft_consensus.cc:374] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:35.093832 18987 raft_consensus.cc:729] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa202c0c845432f9f6ea07bfb413ad6, State: Initialized, Role: FOLLOWER
I20260620 12:23:35.094242 18987 consensus_queue.cc:260] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.094879 18834 heartbeater.cc:503] Master 127.0.98.62:40969 was elected leader, sending a full tablet report...
I20260620 12:23:35.095139 18987 ts_tablet_manager.cc:1428] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 12:23:35.098584 18701 tablet.cc:2369] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:23:35.262145 18969 tablet.cc:2369] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:35.283519 18991 raft_consensus.cc:482] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:35.283741 18991 raft_consensus.cc:504] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.284991 18991 leader_election.cc:288] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0dc571266c947f48d214991526d1570 (127.0.98.1:36983), eaa202c0c845432f9f6ea07bfb413ad6 (127.0.98.2:45319)
I20260620 12:23:35.289321 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.285379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56469 (local address 127.0.98.1:36983)
0620 12:23:35.285509 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.285514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.285954 (+ 440us) server_negotiation.cc:407] Connection header received
0620 12:23:35.286049 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.286052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.286102 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.286209 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:35.286515 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.287144 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.288206 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.288652 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.288753 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.288871 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.289091 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.289186 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:23:35.289354 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.285267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36983 (local address 127.0.98.3:56469)
0620 12:23:35.285835 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.285851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:35.285944 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:35.286242 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:35.286244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:35.286260 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:35.286397 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.286403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.287259 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.287262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.288081 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.288089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.288767 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.288771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.288896 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.288954 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 12:23:35.289027 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 12:23:35.289173 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":211,"thread_start_us":151,"threads_started":1}
I20260620 12:23:35.289397 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.285267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.98.3:54749)
0620 12:23:35.285403 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.285423 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:35.285542 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:35.285986 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:35.285991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:35.286020 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:35.286162 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.286223 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.287030 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.287034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.288084 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.288095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.288901 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.288906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.289078 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.289105 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 12:23:35.289194 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 12:23:35.289267 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":55}
I20260620 12:23:35.289572 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.285388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54749 (local address 127.0.98.2:45319)
0620 12:23:35.285507 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.285511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.285530 (+ 19us) server_negotiation.cc:407] Connection header received
0620 12:23:35.285746 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.285750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.285805 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.285905 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:35.286338 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.286918 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.288255 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.288785 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.288885 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.288949 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.289364 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.289443 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 12:23:35.290112 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4620dd48c5a4d1da5b09fb938d83be0" candidate_uuid: "f5d17ef2c195404e93d163e83330b07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0dc571266c947f48d214991526d1570" is_pre_election: true
I20260620 12:23:35.290110 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4620dd48c5a4d1da5b09fb938d83be0" candidate_uuid: "f5d17ef2c195404e93d163e83330b07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" is_pre_election: true
I20260620 12:23:35.290395 18654 raft_consensus.cc:2454] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d17ef2c195404e93d163e83330b07f in term 0.
I20260620 12:23:35.290411 18788 raft_consensus.cc:2454] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d17ef2c195404e93d163e83330b07f in term 0.
I20260620 12:23:35.290891 18855 leader_election.cc:302] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0dc571266c947f48d214991526d1570, f5d17ef2c195404e93d163e83330b07f; no voters:
I20260620 12:23:35.291148 18991 raft_consensus.cc:2781] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:23:35.291235 18991 raft_consensus.cc:482] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:23:35.291278 18991 raft_consensus.cc:3037] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:35.292521 18991 raft_consensus.cc:504] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.292891 18991 leader_election.cc:288] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [CANDIDATE]: Term 1 election: Requested vote from peers b0dc571266c947f48d214991526d1570 (127.0.98.1:36983), eaa202c0c845432f9f6ea07bfb413ad6 (127.0.98.2:45319)
I20260620 12:23:35.293239 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4620dd48c5a4d1da5b09fb938d83be0" candidate_uuid: "f5d17ef2c195404e93d163e83330b07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0dc571266c947f48d214991526d1570"
I20260620 12:23:35.293361 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4620dd48c5a4d1da5b09fb938d83be0" candidate_uuid: "f5d17ef2c195404e93d163e83330b07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa202c0c845432f9f6ea07bfb413ad6"
I20260620 12:23:35.293390 18654 raft_consensus.cc:3037] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:35.293537 18788 raft_consensus.cc:3037] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:35.294425 18654 raft_consensus.cc:2454] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d17ef2c195404e93d163e83330b07f in term 1.
I20260620 12:23:35.294708 18855 leader_election.cc:302] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0dc571266c947f48d214991526d1570, f5d17ef2c195404e93d163e83330b07f; no voters:
I20260620 12:23:35.294834 18788 raft_consensus.cc:2454] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d17ef2c195404e93d163e83330b07f in term 1.
I20260620 12:23:35.294862 18991 raft_consensus.cc:2781] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:35.295079 18991 raft_consensus.cc:686] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 1 LEADER]: Becoming Leader. State: Replica: f5d17ef2c195404e93d163e83330b07f, State: Running, Role: LEADER
I20260620 12:23:35.295429 18991 consensus_queue.cc:237] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } }
I20260620 12:23:35.298494 18508 catalog_manager.cc:5495] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5d17ef2c195404e93d163e83330b07f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f5d17ef2c195404e93d163e83330b07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5d17ef2c195404e93d163e83330b07f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } health_report { overall_health: UNKNOWN } } }
W20260620 12:23:35.300803 18835 tablet.cc:2369] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:35.357156 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.353600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.0.98.3:32981)
0620 12:23:35.353720 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.353724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.353738 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:23:35.353842 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.353845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.353900 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.353986 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:23:35.354261 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.354758 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.355582 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.355800 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.355892 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.356238 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:23:35.356640 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:23:35.356752 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.356886 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.356949 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:23:35.363014 18654 raft_consensus.cc:1261] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 1 FOLLOWER]: Refusing update from remote peer f5d17ef2c195404e93d163e83330b07f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:35.363014 18788 raft_consensus.cc:1261] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 1 FOLLOWER]: Refusing update from remote peer f5d17ef2c195404e93d163e83330b07f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:35.363749 18991 consensus_queue.cc:1035] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa202c0c845432f9f6ea07bfb413ad6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:35.363957 18995 consensus_queue.cc:1035] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0dc571266c947f48d214991526d1570" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:35.370971 18998 mvcc.cc:205] Tried to move back new op lower bound from 7298900850119782400 to 7298900849852657664. Current Snapshot: MvccSnapshot[applied={T|T < 7298900850119782400}]
I20260620 12:23:35.371881 19000 mvcc.cc:205] Tried to move back new op lower bound from 7298900850119782400 to 7298900849852657664. Current Snapshot: MvccSnapshot[applied={T|T < 7298900850119782400}]
I20260620 12:23:35.381726 18508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:35.381908 18508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:35.384320 18508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76c3485bef5b4f50836190aae242a2c6: Sending DeleteTablet for 3 replicas of tablet d4620dd48c5a4d1da5b09fb938d83be0
I20260620 12:23:35.385105 18902 tablet_service.cc:1514] Processing DeleteTablet for tablet d4620dd48c5a4d1da5b09fb938d83be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47568
I20260620 12:23:35.385273 18768 tablet_service.cc:1514] Processing DeleteTablet for tablet d4620dd48c5a4d1da5b09fb938d83be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49558
I20260620 12:23:35.385648 18634 tablet_service.cc:1514] Processing DeleteTablet for tablet d4620dd48c5a4d1da5b09fb938d83be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55860
I20260620 12:23:35.385668 19008 tablet_replica.cc:330] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: stopping tablet replica
I20260620 12:23:35.385978 19008 raft_consensus.cc:2229] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:23:35.386044 19010 tablet_replica.cc:330] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570: stopping tablet replica
I20260620 12:23:35.386318 19010 raft_consensus.cc:2229] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:35.386364 19008 raft_consensus.cc:2258] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:35.386622 19010 raft_consensus.cc:2258] T d4620dd48c5a4d1da5b09fb938d83be0 P b0dc571266c947f48d214991526d1570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:35.386672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 18571
I20260620 12:23:35.387775 19008 ts_tablet_manager.cc:1905] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:35.390143 19009 tablet_replica.cc:330] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: stopping tablet replica
I20260620 12:23:35.390463 19009 raft_consensus.cc:2229] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:35.390595 19008 ts_tablet_manager.cc:1918] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:35.390673 19008 log.cc:1196] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/d4620dd48c5a4d1da5b09fb938d83be0
I20260620 12:23:35.390746 19009 raft_consensus.cc:2258] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:35.390995 19008 ts_tablet_manager.cc:1939] T d4620dd48c5a4d1da5b09fb938d83be0 P f5d17ef2c195404e93d163e83330b07f: Deleting consensus metadata
I20260620 12:23:35.392248 19009 ts_tablet_manager.cc:1905] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:35.392508 18495 catalog_manager.cc:4841] TS f5d17ef2c195404e93d163e83330b07f (127.0.98.3:32981): tablet d4620dd48c5a4d1da5b09fb938d83be0 (table test-table [id=525382e0ac6b4edca75e0935809d66d8]) successfully deleted
W20260620 12:23:35.395041 18493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36983 (error 108)
I20260620 12:23:35.395702 19009 ts_tablet_manager.cc:1918] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:35.395802 19009 log.cc:1196] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/d4620dd48c5a4d1da5b09fb938d83be0
I20260620 12:23:35.395902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 18705
I20260620 12:23:35.396148 19009 ts_tablet_manager.cc:1939] T d4620dd48c5a4d1da5b09fb938d83be0 P eaa202c0c845432f9f6ea07bfb413ad6: Deleting consensus metadata
I20260620 12:23:35.396632 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.396254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36983 (local address 127.0.0.1:55864)
0620 12:23:35.396456 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.396529 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260620 12:23:35.396817 18493 catalog_manager.cc:4584] TS b0dc571266c947f48d214991526d1570 (127.0.98.1:36983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4620dd48c5a4d1da5b09fb938d83be0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36983: connect: Connection refused (error 111)
I20260620 12:23:35.401826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 18839
I20260620 12:23:35.402197 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.401895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:49572)
0620 12:23:35.402040 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.402105 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 12:23:35.407840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 18477
2026-06-20T12:23:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2471 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T12:23:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:35Z Disabled control of system clock
I20260620 12:23:35.437855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45421
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:23:35.544325 19015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:35.544576 19015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:35.544610 19015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:35.548614 19015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:35.548676 19015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:35.548692 19015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:35.548709 19015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:35.548724 19015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:35.553447 19015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:35.554557 19015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:35.555873 19015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:35.564083 19021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:35.564196 19015 server_base.cc:1034] running on GCE node
W20260620 12:23:35.564092 19023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:35.564126 19020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:35.564841 19015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:35.565950 19015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:35.567104 19015 hybrid_clock.cc:648] HybridClock initialized: now 1781958215567077 us; error 42 us; skew 500 ppm
I20260620 12:23:35.568851 19015 webserver.cc:460] Webserver started at http://127.0.98.62:38841/ using document root <none> and password file <none>
I20260620 12:23:35.569464 19015 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:35.569525 19015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:35.569737 19015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:35.571709 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "b44c6308366f43f287dfa308822d6b64"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "c1f2afde5417ce84b0269a5340ff0eed"
server_key_iv: "e96b3fde9d1c9efa7055e2b44856939f"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.572274 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "b44c6308366f43f287dfa308822d6b64"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "c1f2afde5417ce84b0269a5340ff0eed"
server_key_iv: "e96b3fde9d1c9efa7055e2b44856939f"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.575903 19015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:23:35.578300 19029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:35.579487 19015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:23:35.579624 19015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "b44c6308366f43f287dfa308822d6b64"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "c1f2afde5417ce84b0269a5340ff0eed"
server_key_iv: "e96b3fde9d1c9efa7055e2b44856939f"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.579732 19015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:35.592146 19015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:35.592777 19015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:35.592969 19015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:35.599722 19015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45421
I20260620 12:23:35.599747 19081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45421 every 8 connection(s)
I20260620 12:23:35.600667 19015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:35.602901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 19015
I20260620 12:23:35.603026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:35.603332 392 external_mini_cluster.cc:1428] Setting key ebd885f47e3de4ae9a0cb0796ad524c7
I20260620 12:23:35.604058 19082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:35.609789 19082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64: Bootstrap starting.
I20260620 12:23:35.612038 19082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:35.612213 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.604681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.0.98.62:45421)
0620 12:23:35.605366 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.605382 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.605417 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:23:35.606053 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.606068 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.606437 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.606765 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:35.607232 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.608154 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.609023 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.609533 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.609705 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.610557 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:35.610561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:35.610573 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:35.610937 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:35.610989 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:35.611009 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:35.611116 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.611266 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.611509 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260620 12:23:35.612819 19082 log.cc:824] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:35.614833 19082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64: No bootstrap required, opened a new log
I20260620 12:23:35.618158 19082 raft_consensus.cc:348] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c6308366f43f287dfa308822d6b64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } }
I20260620 12:23:35.618405 19082 raft_consensus.cc:374] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:35.618485 19082 raft_consensus.cc:729] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44c6308366f43f287dfa308822d6b64, State: Initialized, Role: FOLLOWER
I20260620 12:23:35.618902 19082 consensus_queue.cc:260] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c6308366f43f287dfa308822d6b64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } }
I20260620 12:23:35.619051 19082 raft_consensus.cc:388] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:35.619135 19082 raft_consensus.cc:482] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:35.619256 19082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:35.620373 19082 raft_consensus.cc:504] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c6308366f43f287dfa308822d6b64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } }
I20260620 12:23:35.620704 19082 leader_election.cc:302] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b44c6308366f43f287dfa308822d6b64; no voters:
I20260620 12:23:35.620991 19082 leader_election.cc:288] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:35.621137 19087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:35.621374 19087 raft_consensus.cc:686] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [term 1 LEADER]: Becoming Leader. State: Replica: b44c6308366f43f287dfa308822d6b64, State: Running, Role: LEADER
I20260620 12:23:35.621688 19087 consensus_queue.cc:237] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c6308366f43f287dfa308822d6b64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } }
I20260620 12:23:35.622161 19082 sys_catalog.cc:564] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:35.623433 19089 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b44c6308366f43f287dfa308822d6b64. Latest consensus state: current_term: 1 leader_uuid: "b44c6308366f43f287dfa308822d6b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c6308366f43f287dfa308822d6b64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } } }
I20260620 12:23:35.623560 19089 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:35.623533 19088 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b44c6308366f43f287dfa308822d6b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c6308366f43f287dfa308822d6b64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } } }
I20260620 12:23:35.623629 19088 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:35.624670 19096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:35.627363 19096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:35.632642 19096 catalog_manager.cc:1347] Generated new cluster ID: 5d6b58b0989c44b7a4db34066d33ad30
I20260620 12:23:35.632702 19096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:35.650895 19096 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:35.651832 19096 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:35.657426 19096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64: Generated new TSK 0
I20260620 12:23:35.658088 19096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:35.667466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45421
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:23:35.780001 19106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:35.780248 19106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:35.780287 19106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:35.784468 19106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:35.784518 19106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:35.784596 19106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:23:35.789309 19106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:35.790305 19106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:35.791663 19106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:35.801034 19111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:35.801110 19112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:35.801102 19106 server_base.cc:1034] running on GCE node
W20260620 12:23:35.801126 19114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:35.801669 19106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:35.802312 19106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:35.803536 19106 hybrid_clock.cc:648] HybridClock initialized: now 1781958215803512 us; error 37 us; skew 500 ppm
I20260620 12:23:35.805413 19106 webserver.cc:460] Webserver started at http://127.0.98.1:34505/ using document root <none> and password file <none>
I20260620 12:23:35.805963 19106 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:35.806061 19106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:35.806290 19106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:35.808282 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "88a638dabdd9457c81c67ccbfdf7bbf3"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "916a0d24bb8ae2c37f2d86318b729b97"
server_key_iv: "137394b394cbb1def2e65c541619983c"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.808849 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "88a638dabdd9457c81c67ccbfdf7bbf3"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "916a0d24bb8ae2c37f2d86318b729b97"
server_key_iv: "137394b394cbb1def2e65c541619983c"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.812517 19106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:23:35.814798 19120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:35.815963 19106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:35.816090 19106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "88a638dabdd9457c81c67ccbfdf7bbf3"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "916a0d24bb8ae2c37f2d86318b729b97"
server_key_iv: "137394b394cbb1def2e65c541619983c"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.816203 19106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:35.826792 19106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:35.827405 19106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:35.827590 19106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:35.828131 19106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:35.829013 19106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:35.829095 19106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:35.829169 19106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:35.829222 19106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:35.837841 19106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36677
I20260620 12:23:35.837867 19233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36677 every 8 connection(s)
I20260620 12:23:35.838708 19106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:23:35.843422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 19106
I20260620 12:23:35.843557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:23:35.843842 392 external_mini_cluster.cc:1428] Setting key bb40270e91a0c8e95507ac1ba158b1bd
I20260620 12:23:35.845736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45421
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:23:35.848301 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.840485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43593 (local address 127.0.98.62:45421)
0620 12:23:35.840617 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:35.840621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:35.841682 (+ 1061us) server_negotiation.cc:407] Connection header received
0620 12:23:35.843084 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:35.843087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:35.843149 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:35.843253 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:35.844299 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.845082 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.846329 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.846556 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.846667 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.847782 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:35.847786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:35.847788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:35.847804 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:35.847827 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:35.847835 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:35.847924 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:35.848086 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 12:23:35.848163 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:23:35.849157 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:35.840852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45421 (local address 127.0.98.1:43593)
0620 12:23:35.841532 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:35.841571 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:35.842872 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:35.843425 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:35.843432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:35.843552 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:35.844134 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.844147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.845213 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.845222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.846183 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:35.846200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:35.846703 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:35.846707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:35.846888 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:35.847514 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:35.847515 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:35.847554 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:35.847620 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:35.847926 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:35.847937 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:35.847946 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:23:35.848086 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 12:23:35.848359 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":176,"threads_started":1}
I20260620 12:23:35.850667 19234 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45421
I20260620 12:23:35.850941 19234 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:35.851493 19234 heartbeater.cc:511] Master 127.0.98.62:45421 requested a full tablet report, sending...
I20260620 12:23:35.853291 19046 ts_manager.cc:194] Registered new tserver with Master: 88a638dabdd9457c81c67ccbfdf7bbf3 (127.0.98.1:36677)
I20260620 12:23:35.855048 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43593
W20260620 12:23:35.953886 19237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:35.954124 19237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:35.954161 19237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:35.958078 19237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:35.958125 19237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:35.958177 19237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:23:35.962632 19237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:35.963546 19237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:35.964775 19237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:35.973875 19242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:35.973901 19243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:35.973920 19245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:35.974116 19237 server_base.cc:1034] running on GCE node
I20260620 12:23:35.974548 19237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:35.975135 19237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:35.976305 19237 hybrid_clock.cc:648] HybridClock initialized: now 1781958215976287 us; error 37 us; skew 500 ppm
I20260620 12:23:35.978176 19237 webserver.cc:460] Webserver started at http://127.0.98.2:46489/ using document root <none> and password file <none>
I20260620 12:23:35.978755 19237 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:35.978817 19237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:35.979048 19237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:35.981114 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "b162370a044349fba9253c2b7bc0ba0a"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "6322c753a629b548a051ad0c76282b69"
server_key_iv: "6cf6681fba978686896fdcb37eda9861"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.981686 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b162370a044349fba9253c2b7bc0ba0a"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "6322c753a629b548a051ad0c76282b69"
server_key_iv: "6cf6681fba978686896fdcb37eda9861"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.985412 19237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:35.987663 19251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:35.988898 19237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:35.989092 19237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "b162370a044349fba9253c2b7bc0ba0a"
format_stamp: "Formatted at 2026-06-20 12:23:35 on dist-test-slave-9b3m"
server_key: "6322c753a629b548a051ad0c76282b69"
server_key_iv: "6cf6681fba978686896fdcb37eda9861"
server_key_version: "encryptionkey@0"
I20260620 12:23:35.989214 19237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:36.011268 19237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:36.011993 19237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:36.012189 19237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:36.012789 19237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:36.013736 19237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:36.013808 19237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:36.013873 19237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:36.013924 19237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:36.023247 19237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44921
I20260620 12:23:36.023264 19364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44921 every 8 connection(s)
I20260620 12:23:36.024217 19237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:23:36.031977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 19237
I20260620 12:23:36.032090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:23:36.032600 392 external_mini_cluster.cc:1428] Setting key 4908ed798c039f628a7b87265c020143
I20260620 12:23:36.033483 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.026037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35657 (local address 127.0.98.62:45421)
0620 12:23:36.026181 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.026184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.027197 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 12:23:36.028581 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.028585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.028642 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.028746 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:36.029680 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.030207 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.031316 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.031539 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.031651 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.032935 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:36.032939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:36.032941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:36.032956 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:36.032980 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:36.032987 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:36.033082 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.033272 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.033323 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 12:23:36.034336 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.026429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45421 (local address 127.0.98.2:35657)
0620 12:23:36.027049 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:36.027088 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:36.028372 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:36.028913 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:36.028920 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:36.029000 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:36.029538 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.029550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.030337 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.030343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.031194 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.031202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.031689 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.031693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.031860 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.032655 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:36.032658 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:36.032717 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:36.032778 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:36.033081 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:36.033091 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:36.033100 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 12:23:36.033243 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 12:23:36.033494 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260620 12:23:36.034621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45421
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:23:36.035526 19365 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45421
I20260620 12:23:36.035748 19365 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:36.036382 19365 heartbeater.cc:511] Master 127.0.98.62:45421 requested a full tablet report, sending...
I20260620 12:23:36.037567 19046 ts_manager.cc:194] Registered new tserver with Master: b162370a044349fba9253c2b7bc0ba0a (127.0.98.2:44921)
I20260620 12:23:36.038426 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35657
W20260620 12:23:36.146399 19368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:36.146662 19368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:36.146731 19368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:36.150681 19368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:36.150750 19368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:36.150835 19368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:23:36.155409 19368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:36.156423 19368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:36.157872 19368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:36.167263 19374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:36.167289 19373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:36.167351 19368 server_base.cc:1034] running on GCE node
W20260620 12:23:36.167507 19376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:36.167902 19368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:36.168499 19368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:36.169677 19368 hybrid_clock.cc:648] HybridClock initialized: now 1781958216169666 us; error 32 us; skew 500 ppm
I20260620 12:23:36.171561 19368 webserver.cc:460] Webserver started at http://127.0.98.3:42333/ using document root <none> and password file <none>
I20260620 12:23:36.172107 19368 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:36.172194 19368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:36.172422 19368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:36.174422 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "c1fac196b82a445783d2ee63e7ded58b"
format_stamp: "Formatted at 2026-06-20 12:23:36 on dist-test-slave-9b3m"
server_key: "77d65431d116c7f200f12f97c4702e5f"
server_key_iv: "d2b54e0b7d8a28a65b9405468f802522"
server_key_version: "encryptionkey@0"
I20260620 12:23:36.174962 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c1fac196b82a445783d2ee63e7ded58b"
format_stamp: "Formatted at 2026-06-20 12:23:36 on dist-test-slave-9b3m"
server_key: "77d65431d116c7f200f12f97c4702e5f"
server_key_iv: "d2b54e0b7d8a28a65b9405468f802522"
server_key_version: "encryptionkey@0"
I20260620 12:23:36.178438 19368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:36.180636 19382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:36.181841 19368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:36.181968 19368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "c1fac196b82a445783d2ee63e7ded58b"
format_stamp: "Formatted at 2026-06-20 12:23:36 on dist-test-slave-9b3m"
server_key: "77d65431d116c7f200f12f97c4702e5f"
server_key_iv: "d2b54e0b7d8a28a65b9405468f802522"
server_key_version: "encryptionkey@0"
I20260620 12:23:36.182071 19368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:36.196979 19368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:36.197659 19368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:36.197834 19368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:36.198346 19368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:36.199198 19368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:36.199266 19368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:36.199332 19368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:36.199388 19368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:36.207993 19368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38375
I20260620 12:23:36.208016 19495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38375 every 8 connection(s)
I20260620 12:23:36.208796 19368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:23:36.211503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 19368
I20260620 12:23:36.211611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:23:36.211930 392 external_mini_cluster.cc:1428] Setting key 5dfc7e1bfb3cedd82adb05bdee5a0475
I20260620 12:23:36.217948 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.210553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37447 (local address 127.0.98.62:45421)
0620 12:23:36.210695 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.210700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.211901 (+ 1201us) server_negotiation.cc:407] Connection header received
0620 12:23:36.213694 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.213697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.213748 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.213834 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:36.214754 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.215270 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.216324 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.216533 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.216608 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.217551 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:36.217554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:36.217556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:36.217567 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:36.217585 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:36.217593 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:36.217649 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.217778 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.217826 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:23:36.218724 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.210975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45421 (local address 127.0.98.3:37447)
0620 12:23:36.211733 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:36.211788 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:36.213493 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:36.214009 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:36.214020 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:36.214138 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:36.214626 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.214639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.215380 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.215382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.216221 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.216229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.216650 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.216653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.216781 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.217317 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:36.217319 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:36.217356 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:36.217421 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:36.217663 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:36.217669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:36.217673 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:23:36.217762 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 12:23:36.217957 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260620 12:23:36.219771 19496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45421
I20260620 12:23:36.220000 19496 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:36.220485 19496 heartbeater.cc:511] Master 127.0.98.62:45421 requested a full tablet report, sending...
I20260620 12:23:36.221427 19046 ts_manager.cc:194] Registered new tserver with Master: c1fac196b82a445783d2ee63e7ded58b (127.0.98.3:38375)
I20260620 12:23:36.221932 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37447
I20260620 12:23:36.225670 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:23:36.230980 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.227442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.0.98.62:45421)
0620 12:23:36.227574 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.227579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.227757 (+ 178us) server_negotiation.cc:407] Connection header received
0620 12:23:36.227831 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.227833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.227869 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.227933 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:36.228153 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.228704 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.229532 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.229774 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.229851 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.230547 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:36.230549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:36.230551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:36.230561 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:36.230577 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:36.230585 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:36.230680 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.230844 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.230873 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:23:36.233446 19046 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:23:36.236330 19046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:23:36.250231 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.244899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.0.98.2:44921)
0620 12:23:36.245373 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.245391 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.245414 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:23:36.245489 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.245494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.245719 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.245924 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:36.246216 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.247397 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.248858 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.249524 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.249608 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.249746 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.249956 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.250063 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":103,"threads_started":1}
I20260620 12:23:36.250221 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.244386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44921 (local address 127.0.0.1:42562)
0620 12:23:36.244981 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:36.244998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:36.245119 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:36.245904 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:36.245910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:36.245951 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:36.246083 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.246092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.247517 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.247520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.248730 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.248740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.249641 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.249645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.249757 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.249810 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 12:23:36.249917 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 12:23:36.250033 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":50,"spinlock_wait_cycles":11392,"thread_start_us":103,"threads_started":1}
I20260620 12:23:36.250308 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.245685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.98.1:36677)
0620 12:23:36.246224 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.246235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.246257 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:23:36.246328 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.246332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.246494 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.246635 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:36.246848 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.247832 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.249135 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.249706 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.249811 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.250001 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.250057 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.250150 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":111,"threads_started":1}
I20260620 12:23:36.250239 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.245319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36677 (local address 127.0.0.1:43422)
0620 12:23:36.246054 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:36.246069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:36.246149 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:36.246637 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:36.246640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:36.246657 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:36.246728 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.246733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.247967 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.247970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.248999 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.249010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.249836 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.249841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.249945 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.249970 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:23:36.250033 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 12:23:36.250095 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":80,"threads_started":1}
I20260620 12:23:36.251935 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.244204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38375 (local address 127.0.0.1:50210)
0620 12:23:36.244554 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:36.244600 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:36.244738 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:36.245912 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:36.245914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:36.245955 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:36.246093 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.246103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.247285 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.247289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.248733 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.248740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.251274 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.251278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.251637 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.251660 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:23:36.251730 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 12:23:36.251804 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":295,"spinlock_wait_cycles":6400,"thread_start_us":68,"threads_started":1}
I20260620 12:23:36.252416 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.244818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.0.98.3:38375)
0620 12:23:36.245208 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.245222 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.245249 (+ 27us) server_negotiation.cc:407] Connection header received
0620 12:23:36.245326 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.245331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.245746 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.246248 (+ 502us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:36.246347 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.247177 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.250311 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.251132 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.251323 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.251496 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.252207 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.252295 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":177,"threads_started":1}
I20260620 12:23:36.253607 19299 tablet_service.cc:1467] Processing CreateTablet for tablet 0d24cefe19f044daaebff46914e208bc (DEFAULT_TABLE table=test-table [id=3d1ad79c270f45cb8b8feec9d6c4a9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:36.253607 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 0d24cefe19f044daaebff46914e208bc (DEFAULT_TABLE table=test-table [id=3d1ad79c270f45cb8b8feec9d6c4a9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:36.254705 19168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d24cefe19f044daaebff46914e208bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:36.254705 19299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d24cefe19f044daaebff46914e208bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:36.254688 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 0d24cefe19f044daaebff46914e208bc (DEFAULT_TABLE table=test-table [id=3d1ad79c270f45cb8b8feec9d6c4a9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:36.255713 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d24cefe19f044daaebff46914e208bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:36.261337 19513 tablet_bootstrap.cc:492] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Bootstrap starting.
I20260620 12:23:36.262496 19514 tablet_bootstrap.cc:492] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Bootstrap starting.
I20260620 12:23:36.263433 19513 tablet_bootstrap.cc:654] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:36.264135 19513 log.cc:824] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:36.264163 19514 tablet_bootstrap.cc:654] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:36.264720 19515 tablet_bootstrap.cc:492] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3: Bootstrap starting.
I20260620 12:23:36.264971 19514 log.cc:824] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:36.266166 19513 tablet_bootstrap.cc:492] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: No bootstrap required, opened a new log
I20260620 12:23:36.266378 19513 ts_tablet_manager.cc:1397] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:23:36.266785 19515 tablet_bootstrap.cc:654] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:36.266923 19514 tablet_bootstrap.cc:492] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: No bootstrap required, opened a new log
I20260620 12:23:36.267105 19514 ts_tablet_manager.cc:1397] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 12:23:36.267606 19515 log.cc:824] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:36.269513 19515 tablet_bootstrap.cc:492] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3: No bootstrap required, opened a new log
I20260620 12:23:36.269682 19515 ts_tablet_manager.cc:1397] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:23:36.269989 19513 raft_consensus.cc:348] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.270210 19513 raft_consensus.cc:374] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:36.270279 19513 raft_consensus.cc:729] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fac196b82a445783d2ee63e7ded58b, State: Initialized, Role: FOLLOWER
I20260620 12:23:36.270732 19513 consensus_queue.cc:260] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.271349 19496 heartbeater.cc:503] Master 127.0.98.62:45421 was elected leader, sending a full tablet report...
I20260620 12:23:36.271200 19514 raft_consensus.cc:348] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.271438 19514 raft_consensus.cc:374] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:36.271536 19514 raft_consensus.cc:729] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b162370a044349fba9253c2b7bc0ba0a, State: Initialized, Role: FOLLOWER
I20260620 12:23:36.271714 19513 ts_tablet_manager.cc:1428] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:23:36.271937 19514 consensus_queue.cc:260] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.272573 19365 heartbeater.cc:503] Master 127.0.98.62:45421 was elected leader, sending a full tablet report...
I20260620 12:23:36.272841 19514 ts_tablet_manager.cc:1428] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20260620 12:23:36.275095 19366 tablet.cc:2369] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:36.275484 19515 raft_consensus.cc:348] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.275728 19515 raft_consensus.cc:374] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:36.275835 19515 raft_consensus.cc:729] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a638dabdd9457c81c67ccbfdf7bbf3, State: Initialized, Role: FOLLOWER
I20260620 12:23:36.276265 19515 consensus_queue.cc:260] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.276855 19234 heartbeater.cc:503] Master 127.0.98.62:45421 was elected leader, sending a full tablet report...
I20260620 12:23:36.277113 19515 ts_tablet_manager.cc:1428] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 12:23:36.339870 19235 tablet.cc:2369] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 12:23:36.459823 19497 tablet.cc:2369] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:36.567142 19519 raft_consensus.cc:482] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:36.567323 19519 raft_consensus.cc:504] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.568451 19519 leader_election.cc:288] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b162370a044349fba9253c2b7bc0ba0a (127.0.98.2:44921), 88a638dabdd9457c81c67ccbfdf7bbf3 (127.0.98.1:36677)
I20260620 12:23:36.572111 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.568753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53837 (local address 127.0.98.2:44921)
0620 12:23:36.568880 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.568884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.568900 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:23:36.568999 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.569006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.569090 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.569190 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:36.569458 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.570021 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.571184 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.571580 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.571696 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.571755 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.571929 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.571991 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 12:23:36.572113 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.568668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44921 (local address 127.0.98.3:53837)
0620 12:23:36.568784 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:36.568800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:36.568896 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:36.569200 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:36.569204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:36.569221 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:36.569338 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.569344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.570124 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.570129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.571069 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.571078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.571696 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.571699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.571794 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.571820 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 12:23:36.571907 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 12:23:36.571977 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":34}
I20260620 12:23:36.572279 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.568798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59975 (local address 127.0.98.1:36677)
0620 12:23:36.568929 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.568936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.569299 (+ 363us) server_negotiation.cc:407] Connection header received
0620 12:23:36.569447 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.569450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.569499 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.569578 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:36.569835 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.570327 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.571361 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.571737 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.571849 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.571904 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.572101 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.572166 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:23:36.572301 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.568674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36677 (local address 127.0.98.3:59975)
0620 12:23:36.569199 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:36.569210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:36.569334 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:36.569587 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:36.569590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:36.569604 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:36.569729 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.569735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.570430 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.570435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.571240 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:36.571248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.571852 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.571855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:36.571948 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.571976 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 12:23:36.572084 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:23:36.572151 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":168,"thread_start_us":150,"threads_started":1}
I20260620 12:23:36.572988 19319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d24cefe19f044daaebff46914e208bc" candidate_uuid: "c1fac196b82a445783d2ee63e7ded58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b162370a044349fba9253c2b7bc0ba0a" is_pre_election: true
I20260620 12:23:36.572975 19188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d24cefe19f044daaebff46914e208bc" candidate_uuid: "c1fac196b82a445783d2ee63e7ded58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" is_pre_election: true
I20260620 12:23:36.573309 19188 raft_consensus.cc:2454] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fac196b82a445783d2ee63e7ded58b in term 0.
I20260620 12:23:36.573309 19319 raft_consensus.cc:2454] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fac196b82a445783d2ee63e7ded58b in term 0.
I20260620 12:23:36.573803 19384 leader_election.cc:302] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 88a638dabdd9457c81c67ccbfdf7bbf3, b162370a044349fba9253c2b7bc0ba0a, c1fac196b82a445783d2ee63e7ded58b; no voters:
I20260620 12:23:36.574080 19519 raft_consensus.cc:2781] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:23:36.574158 19519 raft_consensus.cc:482] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:23:36.574187 19519 raft_consensus.cc:3037] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:36.575193 19519 raft_consensus.cc:504] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.575515 19519 leader_election.cc:288] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [CANDIDATE]: Term 1 election: Requested vote from peers b162370a044349fba9253c2b7bc0ba0a (127.0.98.2:44921), 88a638dabdd9457c81c67ccbfdf7bbf3 (127.0.98.1:36677)
I20260620 12:23:36.575845 19188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d24cefe19f044daaebff46914e208bc" candidate_uuid: "c1fac196b82a445783d2ee63e7ded58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3"
I20260620 12:23:36.575845 19319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d24cefe19f044daaebff46914e208bc" candidate_uuid: "c1fac196b82a445783d2ee63e7ded58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b162370a044349fba9253c2b7bc0ba0a"
I20260620 12:23:36.575984 19319 raft_consensus.cc:3037] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:36.575984 19188 raft_consensus.cc:3037] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:36.577230 19188 raft_consensus.cc:2454] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fac196b82a445783d2ee63e7ded58b in term 1.
I20260620 12:23:36.577230 19319 raft_consensus.cc:2454] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fac196b82a445783d2ee63e7ded58b in term 1.
I20260620 12:23:36.577554 19384 leader_election.cc:302] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a638dabdd9457c81c67ccbfdf7bbf3, c1fac196b82a445783d2ee63e7ded58b; no voters:
I20260620 12:23:36.577757 19519 raft_consensus.cc:2781] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:36.577999 19519 raft_consensus.cc:686] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 1 LEADER]: Becoming Leader. State: Replica: c1fac196b82a445783d2ee63e7ded58b, State: Running, Role: LEADER
I20260620 12:23:36.578336 19519 consensus_queue.cc:237] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } }
I20260620 12:23:36.581583 19045 catalog_manager.cc:5495] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b reported cstate change: term changed from 0 to 1, leader changed from <none> to c1fac196b82a445783d2ee63e7ded58b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c1fac196b82a445783d2ee63e7ded58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fac196b82a445783d2ee63e7ded58b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 } health_report { overall_health: UNKNOWN } } }
I20260620 12:23:36.613104 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.609586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.0.98.3:38375)
0620 12:23:36.609723 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:36.609727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:36.609743 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:23:36.609862 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:36.609864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:36.609919 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:36.610030 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:23:36.610273 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.610772 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.611646 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:36.611864 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:36.611983 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:36.612267 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:23:36.612622 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:23:36.612773 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:36.612871 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 12:23:36.612903 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:23:36.618718 19188 raft_consensus.cc:1261] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 1 FOLLOWER]: Refusing update from remote peer c1fac196b82a445783d2ee63e7ded58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:36.618718 19319 raft_consensus.cc:1261] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 1 FOLLOWER]: Refusing update from remote peer c1fac196b82a445783d2ee63e7ded58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:36.619445 19523 consensus_queue.cc:1035] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b162370a044349fba9253c2b7bc0ba0a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:36.619683 19519 consensus_queue.cc:1035] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a638dabdd9457c81c67ccbfdf7bbf3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:36.628619 19527 mvcc.cc:205] Tried to move back new op lower bound from 7298900855262502912 to 7298900855107629056. Current Snapshot: MvccSnapshot[applied={T|T < 7298900855262502912}]
I20260620 12:23:36.634007 19045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:36.634181 19045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:36.636552 19045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b44c6308366f43f287dfa308822d6b64: Sending DeleteTablet for 3 replicas of tablet 0d24cefe19f044daaebff46914e208bc
I20260620 12:23:36.637298 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d24cefe19f044daaebff46914e208bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:36 UTC) from {username='slave'} at 127.0.0.1:50210
I20260620 12:23:36.637298 19299 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d24cefe19f044daaebff46914e208bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:36 UTC) from {username='slave'} at 127.0.0.1:42562
I20260620 12:23:36.637342 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d24cefe19f044daaebff46914e208bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:36 UTC) from {username='slave'} at 127.0.0.1:43422
I20260620 12:23:36.637794 19537 tablet_replica.cc:330] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3: stopping tablet replica
I20260620 12:23:36.637887 19535 tablet_replica.cc:330] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: stopping tablet replica
I20260620 12:23:36.638160 19535 raft_consensus.cc:2229] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:23:36.638160 19537 raft_consensus.cc:2229] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:36.638476 19535 raft_consensus.cc:2258] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:36.638520 19537 raft_consensus.cc:2258] T 0d24cefe19f044daaebff46914e208bc P 88a638dabdd9457c81c67ccbfdf7bbf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:36.638665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 19106
I20260620 12:23:36.639745 19535 ts_tablet_manager.cc:1905] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:36.641247 19536 tablet_replica.cc:330] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: stopping tablet replica
I20260620 12:23:36.641530 19536 raft_consensus.cc:2229] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:36.641789 19536 raft_consensus.cc:2258] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:36.642035 19535 ts_tablet_manager.cc:1918] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:36.642155 19535 log.cc:1196] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/0d24cefe19f044daaebff46914e208bc
I20260620 12:23:36.642500 19535 ts_tablet_manager.cc:1939] T 0d24cefe19f044daaebff46914e208bc P c1fac196b82a445783d2ee63e7ded58b: Deleting consensus metadata
I20260620 12:23:36.643250 19536 ts_tablet_manager.cc:1905] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:36.643545 19033 catalog_manager.cc:4841] TS c1fac196b82a445783d2ee63e7ded58b (127.0.98.3:38375): tablet 0d24cefe19f044daaebff46914e208bc (table test-table [id=3d1ad79c270f45cb8b8feec9d6c4a9d8]) successfully deleted
I20260620 12:23:36.646356 19536 ts_tablet_manager.cc:1918] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:36.646456 19536 log.cc:1196] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781957967886955-392-0/minicluster-data/ts-1/wal/wals/0d24cefe19f044daaebff46914e208bc
I20260620 12:23:36.646801 19536 ts_tablet_manager.cc:1939] T 0d24cefe19f044daaebff46914e208bc P b162370a044349fba9253c2b7bc0ba0a: Deleting consensus metadata
I20260620 12:23:36.647976 19030 catalog_manager.cc:4841] TS b162370a044349fba9253c2b7bc0ba0a (127.0.98.2:44921): tablet 0d24cefe19f044daaebff46914e208bc (table test-table [id=3d1ad79c270f45cb8b8feec9d6c4a9d8]) successfully deleted
W20260620 12:23:36.648102 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36677 (error 108)
I20260620 12:23:36.648490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 19237
I20260620 12:23:36.649151 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:36.648848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36677 (local address 127.0.0.1:43438)
0620 12:23:36.648988 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:36.649070 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 12:23:36.649302 19031 catalog_manager.cc:4584] TS 88a638dabdd9457c81c67ccbfdf7bbf3 (127.0.98.1:36677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d24cefe19f044daaebff46914e208bc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36677: connect: Connection refused (error 111)
I20260620 12:23:36.654708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 19368
I20260620 12:23:36.661213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 19015
2026-06-20T12:23:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:36 dist-test-slave-9b3m krb5kdc[19541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:36 dist-test-slave-9b3m krb5kdc[19541](info): set up 2 sockets
Jun 20 12:23:36 dist-test-slave-9b3m krb5kdc[19541](info): commencing operation
krb5kdc: starting...
W20260620 12:23:37.701186 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:37 dist-test-slave-9b3m krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:23:37.818104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:37.925366 19554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:37.925616 19554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:37.925654 19554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:37.929560 19554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:37.929608 19554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:37.929627 19554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:37.929644 19554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:37.929699 19554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:37.934825 19554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:37.935786 19554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:37.936832 19554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:37.945113 19562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:37.945292 19554 server_base.cc:1034] running on GCE node
W20260620 12:23:37.945128 19559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:37.945174 19560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:37.945890 19554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:37.946908 19554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:37.948063 19554 hybrid_clock.cc:648] HybridClock initialized: now 1781958217948049 us; error 30 us; skew 500 ppm
Jun 20 12:23:37 dist-test-slave-9b3m krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:37.950716 19554 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:23:37.951583 19554 webserver.cc:460] Webserver started at http://127.0.98.62:38891/ using document root <none> and password file <none>
I20260620 12:23:37.952131 19554 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:37.952193 19554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:37.952400 19554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:37.954407 19554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "771d07460d344da88872b1f2a3ec0b45"
format_stamp: "Formatted at 2026-06-20 12:23:37 on dist-test-slave-9b3m"
server_key: "948da2f17dcf864c85e2622a31665d41"
server_key_iv: "0193663d4280c20e0166d721681ecabc"
server_key_version: "encryptionkey@0"
I20260620 12:23:37.954953 19554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "771d07460d344da88872b1f2a3ec0b45"
format_stamp: "Formatted at 2026-06-20 12:23:37 on dist-test-slave-9b3m"
server_key: "948da2f17dcf864c85e2622a31665d41"
server_key_iv: "0193663d4280c20e0166d721681ecabc"
server_key_version: "encryptionkey@0"
I20260620 12:23:37.958519 19554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:37.960752 19569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:37.961966 19554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:37.962090 19554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "771d07460d344da88872b1f2a3ec0b45"
format_stamp: "Formatted at 2026-06-20 12:23:37 on dist-test-slave-9b3m"
server_key: "948da2f17dcf864c85e2622a31665d41"
server_key_iv: "0193663d4280c20e0166d721681ecabc"
server_key_version: "encryptionkey@0"
I20260620 12:23:37.962191 19554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:37.998019 19554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:38.000864 19554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:38.001081 19554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:38.008219 19554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40665
I20260620 12:23:38.008242 19621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40665 every 8 connection(s)
I20260620 12:23:38.009143 19554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:38.011850 19622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:38.013774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 19554
I20260620 12:23:38.013911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:38.014225 392 external_mini_cluster.cc:1428] Setting key bea788db57e5ac66afc848001b4c776b
I20260620 12:23:38.018304 19622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45: Bootstrap starting.
I20260620 12:23:38.020431 19622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:38.021224 19622 log.cc:824] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45: Log is configured to *not* fsync() on all Append() calls
Jun 20 12:23:38 dist-test-slave-9b3m krb5kdc[19541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:38.023241 19622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45: No bootstrap required, opened a new log
I20260620 12:23:38.027524 19622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771d07460d344da88872b1f2a3ec0b45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40665 } }
I20260620 12:23:38.027796 19622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:38.027881 19622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771d07460d344da88872b1f2a3ec0b45, State: Initialized, Role: FOLLOWER
I20260620 12:23:38.028381 19622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771d07460d344da88872b1f2a3ec0b45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40665 } }
I20260620 12:23:38.028509 19622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:38.028590 19622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:38.028679 19622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:38.029366 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.015629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.0.98.62:40665)
0620 12:23:38.016047 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.016073 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.016108 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:23:38.016741 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.016757 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.017146 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.017476 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:38.018312 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.019306 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.020184 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.020731 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.020904 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.023642 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:38.023648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:38.023664 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:38.023693 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:38.026226 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.026662 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.026668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.026674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.026814 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.027064 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.027067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.027068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.027551 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:38.027749 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.028103 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.028423 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20260620 12:23:38.029639 19622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771d07460d344da88872b1f2a3ec0b45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40665 } }
I20260620 12:23:38.029968 19622 leader_election.cc:302] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771d07460d344da88872b1f2a3ec0b45; no voters:
I20260620 12:23:38.030277 19622 leader_election.cc:288] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:38.030341 19627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:38.031471 19622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:38.031548 19627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [term 1 LEADER]: Becoming Leader. State: Replica: 771d07460d344da88872b1f2a3ec0b45, State: Running, Role: LEADER
I20260620 12:23:38.031904 19627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771d07460d344da88872b1f2a3ec0b45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40665 } }
I20260620 12:23:38.033710 19628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "771d07460d344da88872b1f2a3ec0b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771d07460d344da88872b1f2a3ec0b45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40665 } } }
I20260620 12:23:38.034134 19628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:38.034516 19629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771d07460d344da88872b1f2a3ec0b45. Latest consensus state: current_term: 1 leader_uuid: "771d07460d344da88872b1f2a3ec0b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771d07460d344da88872b1f2a3ec0b45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40665 } } }
I20260620 12:23:38.034709 19629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45 [sys.catalog]: This master's current role is: LEADER
W20260620 12:23:38.036708 19641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:23:38.036790 19641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:23:38.036867 19642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:38.041962 19642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:38.047741 19642 catalog_manager.cc:1347] Generated new cluster ID: 74fc33153d11466aa59ec6f3ccccac31
I20260620 12:23:38.047806 19642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:38.069516 19642 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:38.070492 19642 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:38.086122 19642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45: Generated new TSK 0
I20260620 12:23:38.086917 19642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 12:23:38.123338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40665
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:38.259865 19648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:38.260129 19648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:38.260176 19648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:38.264514 19648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:38.264567 19648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:38.264624 19648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:23:38.270010 19648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:38.270990 19648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:38.271978 19648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:38.281270 19653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:38.281304 19654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:38.281304 19656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:38.281426 19648 server_base.cc:1034] running on GCE node
I20260620 12:23:38.281915 19648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:38.282496 19648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:38.283663 19648 hybrid_clock.cc:648] HybridClock initialized: now 1781958218283629 us; error 45 us; skew 500 ppm
Jun 20 12:23:38 dist-test-slave-9b3m krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:38.286347 19648 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 12:23:38.287175 19648 webserver.cc:460] Webserver started at http://127.0.98.1:45679/ using document root <none> and password file <none>
I20260620 12:23:38.287669 19648 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:38.287726 19648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:38.287879 19648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:38.289817 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "db3a83b68bc04b3b863a672332bfa498"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "7d72848e9567cbd5835674d7b5e97792"
server_key_iv: "974246e7874c9b94c1f65b2a3fd0b1c6"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.290306 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "db3a83b68bc04b3b863a672332bfa498"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "7d72848e9567cbd5835674d7b5e97792"
server_key_iv: "974246e7874c9b94c1f65b2a3fd0b1c6"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.293761 19648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:38.295900 19663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.297224 19648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:38.297355 19648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "db3a83b68bc04b3b863a672332bfa498"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "7d72848e9567cbd5835674d7b5e97792"
server_key_iv: "974246e7874c9b94c1f65b2a3fd0b1c6"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.297461 19648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:38.311900 19648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:38.314610 19648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:38.314798 19648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:38.315337 19648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:38.316105 19648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:38.316188 19648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.316254 19648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:38.316306 19648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.324978 19648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44377
I20260620 12:23:38.325022 19776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44377 every 8 connection(s)
I20260620 12:23:38.325819 19648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:23:38.330039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 19648
I20260620 12:23:38.330207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:23:38.330543 392 external_mini_cluster.cc:1428] Setting key 5758aea4bf4de1ffa97c5efd9fc35db8
Jun 20 12:23:38 dist-test-slave-9b3m krb5kdc[19541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:38.340062 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.327556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39923 (local address 127.0.98.62:40665)
0620 12:23:38.327701 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.327705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.328522 (+ 817us) server_negotiation.cc:407] Connection header received
0620 12:23:38.329821 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.329824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.329880 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.329960 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:38.331300 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.331838 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.333072 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.333404 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.333516 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.336131 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:38.336134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:38.336136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:38.336160 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:38.337971 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.338453 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.338459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.338462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.338562 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.338882 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.338886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.338889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.339139 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:38.339261 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.339721 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.339857 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 12:23:38.340754 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.327940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40665 (local address 127.0.98.1:39923)
0620 12:23:38.328368 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:38.328408 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:38.329607 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:38.330115 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:38.330122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:38.330568 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:38.331160 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.331177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.331979 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.331982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.332927 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.332937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.333553 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.333557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.333730 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.334300 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:38.334316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:38.335951 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:38.338093 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:38.338097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:38.338108 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:38.338345 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:38.338660 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:38.338662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:38.338664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:38.338788 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:38.339260 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:38.339265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:38.339477 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 12:23:38.339635 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 12:23:38.339858 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260620 12:23:38.341969 19777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40665
I20260620 12:23:38.342196 19777 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:38.342721 19777 heartbeater.cc:511] Master 127.0.98.62:40665 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 12:23:38.344225 19586 ts_manager.cc:194] Registered new tserver with Master: db3a83b68bc04b3b863a672332bfa498 (127.0.98.1:44377)
I20260620 12:23:38.345885 19586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 12:23:38.358418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40665
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:38.466493 19782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:38.466763 19782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:38.466805 19782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:38.470857 19782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:38.470908 19782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:38.471014 19782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:23:38.476135 19782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:38.477252 19782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:38.478292 19782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:38.487330 19787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:38.487358 19790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:38.487331 19788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:38.487520 19782 server_base.cc:1034] running on GCE node
I20260620 12:23:38.487983 19782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:38.488523 19782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:38.489719 19782 hybrid_clock.cc:648] HybridClock initialized: now 1781958218489696 us; error 34 us; skew 500 ppm
Jun 20 12:23:38 dist-test-slave-9b3m krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:38.492328 19782 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 12:23:38.493176 19782 webserver.cc:460] Webserver started at http://127.0.98.2:33965/ using document root <none> and password file <none>
I20260620 12:23:38.493700 19782 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:38.493757 19782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:38.493906 19782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:38.495826 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "125ebf7face24b97ab28880443660d1b"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "c2245c4aeb6519157f44a0d1b8672fa5"
server_key_iv: "9eec3976cae755244ef19f8305d831bd"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.496337 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "125ebf7face24b97ab28880443660d1b"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "c2245c4aeb6519157f44a0d1b8672fa5"
server_key_iv: "9eec3976cae755244ef19f8305d831bd"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.499871 19782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:23:38.502288 19797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.503585 19782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:23:38.503707 19782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "125ebf7face24b97ab28880443660d1b"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "c2245c4aeb6519157f44a0d1b8672fa5"
server_key_iv: "9eec3976cae755244ef19f8305d831bd"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.503809 19782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:38.520761 19782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:38.523986 19782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:38.524183 19782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:38.524734 19782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:38.525563 19782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:38.525633 19782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.525698 19782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:38.525739 19782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.534281 19782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44695
I20260620 12:23:38.534310 19910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44695 every 8 connection(s)
I20260620 12:23:38.535094 19782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 12:23:38 dist-test-slave-9b3m krb5kdc[19541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:38.544243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 19782
I20260620 12:23:38.544358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:23:38.544701 392 external_mini_cluster.cc:1428] Setting key e80e7660c14f333f556e8afb924d058f
I20260620 12:23:38.548563 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.536739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33651 (local address 127.0.98.62:40665)
0620 12:23:38.536861 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.536865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.537634 (+ 769us) server_negotiation.cc:407] Connection header received
0620 12:23:38.538911 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.538915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.538970 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.539074 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:38.540157 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.540684 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.541850 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.542154 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.542276 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.544525 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:38.544529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:38.544534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:38.544564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:38.546544 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.547001 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.547008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.547012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.547105 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.547406 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.547412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.547417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.547671 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:38.547776 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.548357 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.548449 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 12:23:38.549582 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.537021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40665 (local address 127.0.98.2:33651)
0620 12:23:38.537490 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:38.537531 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:38.538710 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:38.539238 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:38.539245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:38.539570 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:38.540017 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.540029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.540807 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.540810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.541725 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.541736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.542308 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.542312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.542476 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.542946 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:38.542961 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:38.544312 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:38.546654 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:38.546658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:38.546668 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:38.546906 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:38.547198 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:38.547200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:38.547201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:38.547318 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:38.547782 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:38.547791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:38.548104 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 12:23:38.548314 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 12:23:38.548643 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260620 12:23:38.550783 19911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40665
I20260620 12:23:38.551016 19911 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:38.551491 19911 heartbeater.cc:511] Master 127.0.98.62:40665 requested a full tablet report, sending...
I20260620 12:23:38.552568 19586 ts_manager.cc:194] Registered new tserver with Master: 125ebf7face24b97ab28880443660d1b (127.0.98.2:44695)
I20260620 12:23:38.553388 19586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33651
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 12:23:38.571300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40665
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:38.678449 19916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:38.678705 19916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:38.678778 19916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:38.682752 19916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:38.682825 19916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:38.682911 19916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:23:38.687888 19916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:38.688848 19916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:38.689848 19916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:38.699177 19922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:38.699115 19924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:38.699213 19921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:38.699474 19916 server_base.cc:1034] running on GCE node
I20260620 12:23:38.699836 19916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:38.700470 19916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:38.701692 19916 hybrid_clock.cc:648] HybridClock initialized: now 1781958218701653 us; error 45 us; skew 500 ppm
Jun 20 12:23:38 dist-test-slave-9b3m krb5kdc[19541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:38.704653 19916 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 12:23:38.705540 19916 webserver.cc:460] Webserver started at http://127.0.98.3:34353/ using document root <none> and password file <none>
I20260620 12:23:38.706080 19916 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:38.706180 19916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:38.706401 19916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:38.708438 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "cfec6ea3b3cc49429648945f66e9b4c6"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "603afd4578dae414b606548358724af1"
server_key_iv: "c4d461b1d55f48635a26bdabd2bf8168"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.708973 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cfec6ea3b3cc49429648945f66e9b4c6"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "603afd4578dae414b606548358724af1"
server_key_iv: "c4d461b1d55f48635a26bdabd2bf8168"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.712369 19916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 12:23:38.714675 19931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.715811 19916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:23:38.715938 19916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "cfec6ea3b3cc49429648945f66e9b4c6"
format_stamp: "Formatted at 2026-06-20 12:23:38 on dist-test-slave-9b3m"
server_key: "603afd4578dae414b606548358724af1"
server_key_iv: "c4d461b1d55f48635a26bdabd2bf8168"
server_key_version: "encryptionkey@0"
I20260620 12:23:38.716040 19916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:38.730334 19916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:38.733165 19916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:38.733364 19916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:38.733911 19916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:38.734696 19916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:38.734766 19916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.734833 19916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:38.734886 19916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:38.743742 19916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41207
I20260620 12:23:38.743770 20044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41207 every 8 connection(s)
I20260620 12:23:38.744581 19916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:23:38.747011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 19916
I20260620 12:23:38.747169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:23:38.747471 392 external_mini_cluster.cc:1428] Setting key 4a10d76f52f0ce3e9c2c7ea9725860db
Jun 20 12:23:38 dist-test-slave-9b3m krb5kdc[19541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:38.757758 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.746314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41037 (local address 127.0.98.62:40665)
0620 12:23:38.746454 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.746457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.747260 (+ 803us) server_negotiation.cc:407] Connection header received
0620 12:23:38.748946 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.748948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.748989 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.749100 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:38.750369 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.750897 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.752054 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.752272 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.752349 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.754618 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:38.754621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:38.754623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:38.754646 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:38.756047 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.756480 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.756483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.756484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.756554 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.756826 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.756828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.756829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.756987 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:38.757119 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.757543 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.757639 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 12:23:38.758514 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.746673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40665 (local address 127.0.98.3:41037)
0620 12:23:38.747116 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:38.747164 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:38.748704 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:38.749274 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:38.749282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:38.749780 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:38.750244 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.750257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.751020 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.751022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.751951 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.751960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.752394 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.752397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.752535 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.753079 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:23:38.753094 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:38.754477 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:38.756138 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:38.756141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:38.756152 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:38.756396 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:38.756629 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:23:38.756631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:23:38.756632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:23:38.756751 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:23:38.757122 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:38.757127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:38.757343 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 12:23:38.757497 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 12:23:38.757726 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":136,"threads_started":1}
I20260620 12:23:38.759635 20045 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40665
I20260620 12:23:38.759864 20045 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:38.760408 20045 heartbeater.cc:511] Master 127.0.98.62:40665 requested a full tablet report, sending...
I20260620 12:23:38.761425 19586 ts_manager.cc:194] Registered new tserver with Master: cfec6ea3b3cc49429648945f66e9b4c6 (127.0.98.3:41207)
I20260620 12:23:38.761924 19586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41037
I20260620 12:23:38.773989 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:23:38.783347 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.775941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.0.98.62:40665)
0620 12:23:38.776096 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.776100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.776164 (+ 64us) server_negotiation.cc:407] Connection header received
0620 12:23:38.776276 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.776278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.776319 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.776388 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:38.777118 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.777705 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.778517 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.778770 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.778853 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.780137 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:38.780140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:38.780142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:38.780165 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:38.781806 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.782171 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.782174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.782175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.782271 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:38.782524 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:38.782526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:38.782527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:38.782743 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:38.782851 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.783016 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.783108 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 12:23:38.786171 19586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:23:38.789149 19586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:23:38.803718 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.797467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41207 (local address 127.0.0.1:37800)
0620 12:23:38.798873 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:38.798927 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:38.799067 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:38.799809 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:38.799812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:38.799839 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:38.799998 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.800004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.801123 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.801127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.802381 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.802391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.803178 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.803183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.803288 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.803322 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 12:23:38.803436 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 12:23:38.803556 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":49,"spinlock_wait_cycles":8704,"thread_start_us":159,"threads_started":1}
I20260620 12:23:38.803721 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.797574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.0.98.3:41207)
0620 12:23:38.798183 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.798195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.799008 (+ 813us) server_negotiation.cc:407] Connection header received
0620 12:23:38.799322 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.799327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.799472 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.799632 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:38.800138 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.800919 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.802505 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.803054 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.803209 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.803354 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.803464 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.803564 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":122,"threads_started":1}
I20260620 12:23:38.805387 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.797753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.0.98.2:44695)
0620 12:23:38.798379 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.798390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.799012 (+ 622us) server_negotiation.cc:407] Connection header received
0620 12:23:38.799243 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.799266 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.799473 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.799653 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:38.800125 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.801421 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.803383 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.804559 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.804667 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.804809 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.805125 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.805228 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":123,"threads_started":1}
I20260620 12:23:38.805389 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.798444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.98.1:44377)
0620 12:23:38.798775 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.798791 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.799221 (+ 430us) server_negotiation.cc:407] Connection header received
0620 12:23:38.799310 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.799322 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.799475 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.799653 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:38.799962 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.801472 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.803413 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.804285 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.804397 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.804535 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.805146 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.805236 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":136,"threads_started":1}
I20260620 12:23:38.805050 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.798276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44377 (local address 127.0.0.1:49668)
0620 12:23:38.798909 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:38.798926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:38.799059 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:38.799637 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:38.799644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:38.799667 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:38.799820 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.799826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.801721 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.801724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.803212 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.803221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.804608 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.804629 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.804723 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.804748 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:23:38.804823 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 12:23:38.804904 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":142,"spinlock_wait_cycles":121600,"thread_start_us":93,"threads_started":1}
I20260620 12:23:38.805832 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.797571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44695 (local address 127.0.0.1:35104)
0620 12:23:38.798881 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:38.798929 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:38.799099 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:38.799809 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:38.799812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:38.799833 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:38.799987 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.799994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.801590 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.801593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.803146 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.803159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.804833 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.804837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.804926 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.804948 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 12:23:38.805059 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 12:23:38.805130 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":141,"spinlock_wait_cycles":968320,"thread_start_us":114,"threads_started":1}
I20260620 12:23:38.807083 19979 tablet_service.cc:1467] Processing CreateTablet for tablet 729bcc53737b4588b3fb5923c2727eec (DEFAULT_TABLE table=test-table [id=cad3335cfb344ba9a0fffa6e2fb855e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:38.808218 19979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729bcc53737b4588b3fb5923c2727eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:38.808439 19711 tablet_service.cc:1467] Processing CreateTablet for tablet 729bcc53737b4588b3fb5923c2727eec (DEFAULT_TABLE table=test-table [id=cad3335cfb344ba9a0fffa6e2fb855e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:38.808432 19845 tablet_service.cc:1467] Processing CreateTablet for tablet 729bcc53737b4588b3fb5923c2727eec (DEFAULT_TABLE table=test-table [id=cad3335cfb344ba9a0fffa6e2fb855e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:38.809480 19711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729bcc53737b4588b3fb5923c2727eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:38.809540 19845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 729bcc53737b4588b3fb5923c2727eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:38.815426 20062 tablet_bootstrap.cc:492] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b: Bootstrap starting.
I20260620 12:23:38.816578 20063 tablet_bootstrap.cc:492] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Bootstrap starting.
I20260620 12:23:38.817375 20064 tablet_bootstrap.cc:492] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498: Bootstrap starting.
I20260620 12:23:38.817767 20062 tablet_bootstrap.cc:654] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:38.818636 20063 tablet_bootstrap.cc:654] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:38.819044 20062 log.cc:824] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:38.819437 20064 tablet_bootstrap.cc:654] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:38.820052 20063 log.cc:824] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:38.820643 20064 log.cc:824] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:38.821322 20062 tablet_bootstrap.cc:492] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b: No bootstrap required, opened a new log
I20260620 12:23:38.821535 20062 ts_tablet_manager.cc:1397] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:23:38.822001 20063 tablet_bootstrap.cc:492] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: No bootstrap required, opened a new log
I20260620 12:23:38.822196 20063 ts_tablet_manager.cc:1397] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:23:38.822639 20064 tablet_bootstrap.cc:492] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498: No bootstrap required, opened a new log
I20260620 12:23:38.822815 20064 ts_tablet_manager.cc:1397] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 12:23:38.824923 20063 raft_consensus.cc:348] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.825227 20063 raft_consensus.cc:374] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:38.825284 20063 raft_consensus.cc:729] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfec6ea3b3cc49429648945f66e9b4c6, State: Initialized, Role: FOLLOWER
I20260620 12:23:38.825728 20063 consensus_queue.cc:260] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.826349 20045 heartbeater.cc:503] Master 127.0.98.62:40665 was elected leader, sending a full tablet report...
I20260620 12:23:38.826673 20063 ts_tablet_manager.cc:1428] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 12:23:38.826519 20062 raft_consensus.cc:348] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.826898 20062 raft_consensus.cc:374] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:38.826947 20062 raft_consensus.cc:729] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125ebf7face24b97ab28880443660d1b, State: Initialized, Role: FOLLOWER
I20260620 12:23:38.826864 20064 raft_consensus.cc:348] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.827101 20064 raft_consensus.cc:374] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:38.827153 20064 raft_consensus.cc:729] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3a83b68bc04b3b863a672332bfa498, State: Initialized, Role: FOLLOWER
I20260620 12:23:38.827395 20062 consensus_queue.cc:260] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.827527 20064 consensus_queue.cc:260] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.828050 19911 heartbeater.cc:503] Master 127.0.98.62:40665 was elected leader, sending a full tablet report...
I20260620 12:23:38.828366 20062 ts_tablet_manager.cc:1428] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 12:23:38.828366 20064 ts_tablet_manager.cc:1428] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 12:23:38.828667 19777 heartbeater.cc:503] Master 127.0.98.62:40665 was elected leader, sending a full tablet report...
I20260620 12:23:38.954937 20069 raft_consensus.cc:482] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:38.955130 20069 raft_consensus.cc:504] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.956229 20069 leader_election.cc:288] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfec6ea3b3cc49429648945f66e9b4c6 (127.0.98.3:41207), db3a83b68bc04b3b863a672332bfa498 (127.0.98.1:44377)
I20260620 12:23:38.959815 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.956488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44377 (local address 127.0.98.2:48951)
0620 12:23:38.956623 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:38.956640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:38.956745 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:38.957061 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:38.957065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:38.957083 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:38.957202 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.957208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.957951 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.957955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.958739 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.958747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.959385 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.959389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.959498 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.959516 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 12:23:38.959606 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 12:23:38.959678 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 12:23:38.959815 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.956598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48951 (local address 127.0.98.1:44377)
0620 12:23:38.956795 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.956799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.956815 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:23:38.956884 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.956886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.956925 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.957024 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:38.957316 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.957838 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.958863 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.959263 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.959380 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.959442 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.959621 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.959694 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 12:23:38.960027 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.956624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53579 (local address 127.0.98.3:41207)
0620 12:23:38.956755 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.956760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.956993 (+ 233us) server_negotiation.cc:407] Connection header received
0620 12:23:38.957162 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.957165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.957209 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.957295 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:38.957570 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.958042 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.959043 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.959443 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.959560 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.959617 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.959809 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.959877 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:23:38.960489 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.956488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41207 (local address 127.0.98.2:53579)
0620 12:23:38.956911 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:38.956921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:38.957025 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:38.957316 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:38.957319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:38.957334 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:38.957461 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.957467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.958150 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.958155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.958922 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:38.958930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.959565 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.959568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:38.959675 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.959700 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 12:23:38.959792 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 12:23:38.959860 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":116,"thread_start_us":117,"threads_started":1}
I20260620 12:23:38.960542 19731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729bcc53737b4588b3fb5923c2727eec" candidate_uuid: "125ebf7face24b97ab28880443660d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3a83b68bc04b3b863a672332bfa498" is_pre_election: true
I20260620 12:23:38.960860 19731 raft_consensus.cc:2454] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125ebf7face24b97ab28880443660d1b in term 0.
I20260620 12:23:38.961208 19999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729bcc53737b4588b3fb5923c2727eec" candidate_uuid: "125ebf7face24b97ab28880443660d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" is_pre_election: true
I20260620 12:23:38.961442 19799 leader_election.cc:302] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125ebf7face24b97ab28880443660d1b, db3a83b68bc04b3b863a672332bfa498; no voters:
I20260620 12:23:38.961494 19999 raft_consensus.cc:2454] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125ebf7face24b97ab28880443660d1b in term 0.
I20260620 12:23:38.961694 20069 raft_consensus.cc:2781] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:23:38.961805 20069 raft_consensus.cc:482] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:23:38.961866 20069 raft_consensus.cc:3037] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:38.962877 20069 raft_consensus.cc:504] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.963239 20069 leader_election.cc:288] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [CANDIDATE]: Term 1 election: Requested vote from peers cfec6ea3b3cc49429648945f66e9b4c6 (127.0.98.3:41207), db3a83b68bc04b3b863a672332bfa498 (127.0.98.1:44377)
I20260620 12:23:38.963560 19999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729bcc53737b4588b3fb5923c2727eec" candidate_uuid: "125ebf7face24b97ab28880443660d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfec6ea3b3cc49429648945f66e9b4c6"
I20260620 12:23:38.963560 19731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "729bcc53737b4588b3fb5923c2727eec" candidate_uuid: "125ebf7face24b97ab28880443660d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3a83b68bc04b3b863a672332bfa498"
I20260620 12:23:38.963681 19731 raft_consensus.cc:3037] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:38.963681 19999 raft_consensus.cc:3037] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:38.964929 19999 raft_consensus.cc:2454] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125ebf7face24b97ab28880443660d1b in term 1.
I20260620 12:23:38.964929 19731 raft_consensus.cc:2454] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125ebf7face24b97ab28880443660d1b in term 1.
I20260620 12:23:38.965243 19799 leader_election.cc:302] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125ebf7face24b97ab28880443660d1b, db3a83b68bc04b3b863a672332bfa498; no voters:
I20260620 12:23:38.965413 20069 raft_consensus.cc:2781] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:38.965629 20069 raft_consensus.cc:686] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 1 LEADER]: Becoming Leader. State: Replica: 125ebf7face24b97ab28880443660d1b, State: Running, Role: LEADER
I20260620 12:23:38.965967 20069 consensus_queue.cc:237] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } }
I20260620 12:23:38.969344 19585 catalog_manager.cc:5495] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 125ebf7face24b97ab28880443660d1b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "125ebf7face24b97ab28880443660d1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125ebf7face24b97ab28880443660d1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } health_report { overall_health: UNKNOWN } } }
I20260620 12:23:38.991724 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:38.988030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.98.2:44695)
0620 12:23:38.988204 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:38.988209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:38.988224 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:23:38.988388 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:38.988391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:38.988446 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:38.988558 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:23:38.988827 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.989378 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.990255 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:38.990484 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:38.990600 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:38.990907 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:23:38.991307 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:23:38.991441 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:38.991505 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 12:23:38.991559 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 12:23:38.995527 20046 tablet.cc:2369] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:38.997623 19999 raft_consensus.cc:1261] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 1 FOLLOWER]: Refusing update from remote peer 125ebf7face24b97ab28880443660d1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:38.997625 19731 raft_consensus.cc:1261] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 1 FOLLOWER]: Refusing update from remote peer 125ebf7face24b97ab28880443660d1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:38.998318 20072 consensus_queue.cc:1035] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3a83b68bc04b3b863a672332bfa498" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:38.998486 20069 consensus_queue.cc:1035] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfec6ea3b3cc49429648945f66e9b4c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:39.006735 20076 mvcc.cc:205] Tried to move back new op lower bound from 7298900865006501888 to 7298900864887332864. Current Snapshot: MvccSnapshot[applied={T|T < 7298900865006501888}]
I20260620 12:23:39.019249 19586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:39.019415 19586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:39.021765 19586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 771d07460d344da88872b1f2a3ec0b45: Sending DeleteTablet for 3 replicas of tablet 729bcc53737b4588b3fb5923c2727eec
I20260620 12:23:39.022498 19979 tablet_service.cc:1514] Processing DeleteTablet for tablet 729bcc53737b4588b3fb5923c2727eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37800
I20260620 12:23:39.022857 19711 tablet_service.cc:1514] Processing DeleteTablet for tablet 729bcc53737b4588b3fb5923c2727eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49668
I20260620 12:23:39.023288 20086 tablet_replica.cc:330] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498: stopping tablet replica
I20260620 12:23:39.023458 19845 tablet_service.cc:1514] Processing DeleteTablet for tablet 729bcc53737b4588b3fb5923c2727eec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35104
I20260620 12:23:39.023594 20086 raft_consensus.cc:2229] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:39.023900 20086 raft_consensus.cc:2258] T 729bcc53737b4588b3fb5923c2727eec P db3a83b68bc04b3b863a672332bfa498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:39.023954 20085 tablet_replica.cc:330] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: stopping tablet replica
I20260620 12:23:39.024322 20085 raft_consensus.cc:2229] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:39.024379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 19648
I20260620 12:23:39.026723 20085 raft_consensus.cc:2258] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:39.027596 20087 tablet_replica.cc:330] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b: stopping tablet replica
I20260620 12:23:39.027874 20087 raft_consensus.cc:2229] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:23:39.028227 20087 raft_consensus.cc:2258] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:39.028414 20085 ts_tablet_manager.cc:1905] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:39.029731 20087 ts_tablet_manager.cc:1905] T 729bcc53737b4588b3fb5923c2727eec P 125ebf7face24b97ab28880443660d1b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 12:23:39.031144 19571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44377 (error 108)
I20260620 12:23:39.031332 20085 ts_tablet_manager.cc:1918] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:39.031491 20085 log.cc:1196] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/729bcc53737b4588b3fb5923c2727eec
I20260620 12:23:39.031975 20085 ts_tablet_manager.cc:1939] T 729bcc53737b4588b3fb5923c2727eec P cfec6ea3b3cc49429648945f66e9b4c6: Deleting consensus metadata
I20260620 12:23:39.031984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 19782
I20260620 12:23:39.037253 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.036879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44377 (local address 127.0.0.1:49680)
0620 12:23:39.037069 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:39.037153 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 12:23:39.037492 19571 catalog_manager.cc:4584] TS db3a83b68bc04b3b863a672332bfa498 (127.0.98.1:44377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 729bcc53737b4588b3fb5923c2727eec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44377: connect: Connection refused (error 111)
I20260620 12:23:39.037624 19573 catalog_manager.cc:4841] TS cfec6ea3b3cc49429648945f66e9b4c6 (127.0.98.3:41207): tablet 729bcc53737b4588b3fb5923c2727eec (table test-table [id=cad3335cfb344ba9a0fffa6e2fb855e2]) successfully deleted
I20260620 12:23:39.038182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 19916
I20260620 12:23:39.038964 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.038314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44695 (local address 127.0.0.1:35130)
0620 12:23:39.038824 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:39.038883 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425}
I20260620 12:23:39.044138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 19554
2026-06-20T12:23:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2384 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T12:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:39Z Disabled control of system clock
I20260620 12:23:39.074914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41805
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:23:39.183300 20092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:39.183578 20092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:39.183645 20092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:39.187547 20092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:39.187631 20092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:39.187661 20092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:39.187697 20092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:39.187731 20092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:39.192308 20092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:39.193290 20092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:39.194559 20092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:39.202483 20097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:39.202489 20098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:39.202502 20100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:39.202765 20092 server_base.cc:1034] running on GCE node
I20260620 12:23:39.203326 20092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:39.204416 20092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:39.205597 20092 hybrid_clock.cc:648] HybridClock initialized: now 1781958219205565 us; error 46 us; skew 500 ppm
I20260620 12:23:39.207345 20092 webserver.cc:460] Webserver started at http://127.0.98.62:41989/ using document root <none> and password file <none>
I20260620 12:23:39.207975 20092 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:39.208065 20092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:39.208287 20092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:39.210323 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "d1d8ac75397115c47960628647a762a5"
server_key_iv: "eb1ae5e050cc6a4b08141e93fac87668"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.210865 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "d1d8ac75397115c47960628647a762a5"
server_key_iv: "eb1ae5e050cc6a4b08141e93fac87668"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.214306 20092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:39.216614 20106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.217833 20092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:39.217968 20092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "d1d8ac75397115c47960628647a762a5"
server_key_iv: "eb1ae5e050cc6a4b08141e93fac87668"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.218070 20092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:39.229463 20092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:39.230109 20092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:39.230296 20092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:39.236668 20092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41805
I20260620 12:23:39.236689 20158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41805 every 8 connection(s)
I20260620 12:23:39.237618 20092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:39.240214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 20092
I20260620 12:23:39.240357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:39.240641 20159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:39.240661 392 external_mini_cluster.cc:1428] Setting key fbf2865f135b3fee534a48ac6d8d488f
I20260620 12:23:39.247628 20159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e: Bootstrap starting.
I20260620 12:23:39.250262 20159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:39.250773 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.242058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.0.98.62:41805)
0620 12:23:39.242702 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:39.242720 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:39.242771 (+ 51us) server_negotiation.cc:407] Connection header received
0620 12:23:39.243446 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:39.243469 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:39.243966 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:39.244323 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:39.244757 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.246227 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.247115 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.247784 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.247931 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.248756 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:39.248763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:39.248782 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:39.249285 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:39.249354 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:39.249379 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:39.249473 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:39.249580 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 12:23:39.249838 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260620 12:23:39.251088 20159 log.cc:824] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:39.252983 20159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e: No bootstrap required, opened a new log
I20260620 12:23:39.256464 20159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } }
I20260620 12:23:39.256706 20159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:39.256760 20159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b6e6d5e86e457b9c3e9bbfdc45bb5e, State: Initialized, Role: FOLLOWER
I20260620 12:23:39.257213 20159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } }
I20260620 12:23:39.257369 20159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:39.257445 20159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:39.257570 20159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:39.258390 20159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } }
I20260620 12:23:39.258718 20159 leader_election.cc:302] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11b6e6d5e86e457b9c3e9bbfdc45bb5e; no voters:
I20260620 12:23:39.259048 20159 leader_election.cc:288] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:39.259156 20164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:39.259398 20164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [term 1 LEADER]: Becoming Leader. State: Replica: 11b6e6d5e86e457b9c3e9bbfdc45bb5e, State: Running, Role: LEADER
I20260620 12:23:39.259677 20164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } }
I20260620 12:23:39.260187 20159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:39.261327 20165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } }
I20260620 12:23:39.261353 20166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11b6e6d5e86e457b9c3e9bbfdc45bb5e. Latest consensus state: current_term: 1 leader_uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b6e6d5e86e457b9c3e9bbfdc45bb5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } }
I20260620 12:23:39.261445 20165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:39.261469 20166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:39.261962 20173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:39.265087 20173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:39.270706 20173 catalog_manager.cc:1347] Generated new cluster ID: e6922ab28af447a18bf246f6c5bef161
I20260620 12:23:39.270784 20173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:39.280038 20173 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:39.281001 20173 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:39.288496 20173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e: Generated new TSK 0
I20260620 12:23:39.289168 20173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:39.293279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--builtin_ntp_servers=127.0.98.20:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 12:23:39.403121 20183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:39.403393 20183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:39.403462 20183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:39.407559 20183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:39.407639 20183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:39.407729 20183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 12:23:39.412588 20183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:39.413688 20183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:39.415014 20183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:39.424399 20191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:39.424399 20189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:39.424444 20188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:39.424712 20183 server_base.cc:1034] running on GCE node
I20260620 12:23:39.425223 20183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:39.425874 20183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:39.427060 20183 hybrid_clock.cc:648] HybridClock initialized: now 1781958219427031 us; error 42 us; skew 500 ppm
I20260620 12:23:39.428786 20183 webserver.cc:460] Webserver started at http://127.0.98.1:38899/ using document root <none> and password file <none>
I20260620 12:23:39.429420 20183 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:39.429510 20183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:39.429736 20183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:39.431684 20183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/data/instance:
uuid: "6b369fc8e13e4ed6b1d420642ba28f5b"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "027a7058c2f9b2887d0f8f8499deb92d"
server_key_iv: "f4e89dd8682962022c70d38fdacdaeb3"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.432250 20183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6b369fc8e13e4ed6b1d420642ba28f5b"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "027a7058c2f9b2887d0f8f8499deb92d"
server_key_iv: "f4e89dd8682962022c70d38fdacdaeb3"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.435788 20183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 12:23:39.438462 20197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.439961 20183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 12:23:39.440094 20183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/wal
uuid: "6b369fc8e13e4ed6b1d420642ba28f5b"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "027a7058c2f9b2887d0f8f8499deb92d"
server_key_iv: "f4e89dd8682962022c70d38fdacdaeb3"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.440207 20183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:39.469843 20183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:39.470534 20183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:39.470750 20183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:39.471338 20183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:39.472254 20183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:39.472327 20183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.472398 20183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:39.472441 20183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.481441 20183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41559
I20260620 12:23:39.481487 20310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41559 every 8 connection(s)
I20260620 12:23:39.482357 20183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/data/info.pb
I20260620 12:23:39.491108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 20183
I20260620 12:23:39.491254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-0/wal/instance
I20260620 12:23:39.491469 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.484114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38165 (local address 127.0.98.62:41805)
0620 12:23:39.484284 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:39.484288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:39.485363 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 12:23:39.486729 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:39.486732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:39.486792 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:39.486884 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:39.487767 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.488315 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.489506 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.489735 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.489846 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.490981 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:39.490990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:39.490992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:39.491002 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:39.491020 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:39.491027 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:39.491105 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:39.491270 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 12:23:39.491330 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:23:39.491592 392 external_mini_cluster.cc:1428] Setting key 28505a72e8d398a25725a5aeb3f49307
I20260620 12:23:39.492358 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.484510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:38165)
0620 12:23:39.485228 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:39.485269 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:39.486528 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:39.487061 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:39.487067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:39.487152 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:39.487623 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.487636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.488451 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.488454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.489383 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.489392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.489883 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.489887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.490048 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.490698 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:39.490699 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:39.490734 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:39.490814 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:39.491107 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:39.491118 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:39.491126 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 12:23:39.491262 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 12:23:39.491516 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":107,"threads_started":1}
I20260620 12:23:39.493512 20311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260620 12:23:39.493640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--builtin_ntp_servers=127.0.98.20:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:23:39.493748 20311 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:39.494215 20311 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260620 12:23:39.495584 20123 ts_manager.cc:194] Registered new tserver with Master: 6b369fc8e13e4ed6b1d420642ba28f5b (127.0.98.1:41559)
I20260620 12:23:39.496697 20123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38165
W20260620 12:23:39.601925 20314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:39.602162 20314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:39.602198 20314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:39.606290 20314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:39.606338 20314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:39.606423 20314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 12:23:39.611156 20314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:39.612154 20314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:39.613487 20314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:39.622937 20322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:39.622956 20319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:39.622989 20320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:39.623286 20314 server_base.cc:1034] running on GCE node
I20260620 12:23:39.623726 20314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:39.624313 20314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:39.625463 20314 hybrid_clock.cc:648] HybridClock initialized: now 1781958219625434 us; error 42 us; skew 500 ppm
I20260620 12:23:39.630931 20314 webserver.cc:460] Webserver started at http://127.0.98.2:46817/ using document root <none> and password file <none>
I20260620 12:23:39.631496 20314 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:39.631556 20314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:39.631762 20314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:39.633810 20314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/data/instance:
uuid: "d74d706eb2604a29bc060308e0fe2819"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "bf48adac8e36de4f417b46044fc14dc5"
server_key_iv: "e814a669ec53ca6f0b52a3123cbce23e"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.634370 20314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d74d706eb2604a29bc060308e0fe2819"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "bf48adac8e36de4f417b46044fc14dc5"
server_key_iv: "e814a669ec53ca6f0b52a3123cbce23e"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.638028 20314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:39.640354 20328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.641570 20314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:39.641688 20314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/wal
uuid: "d74d706eb2604a29bc060308e0fe2819"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "bf48adac8e36de4f417b46044fc14dc5"
server_key_iv: "e814a669ec53ca6f0b52a3123cbce23e"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.641788 20314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:39.669598 20314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:39.670290 20314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:39.670495 20314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:39.671103 20314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:39.672035 20314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:39.672082 20314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.672158 20314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:39.672201 20314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.681198 20314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41029
I20260620 12:23:39.681226 20441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41029 every 8 connection(s)
I20260620 12:23:39.682085 20314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/data/info.pb
I20260620 12:23:39.689647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 20314
I20260620 12:23:39.689813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-1/wal/instance
I20260620 12:23:39.690094 392 external_mini_cluster.cc:1428] Setting key 95628786a41cf4656b516c2e65eb67ef
I20260620 12:23:39.691047 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.683846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45033 (local address 127.0.98.62:41805)
0620 12:23:39.683981 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:39.683985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:39.684934 (+ 949us) server_negotiation.cc:407] Connection header received
0620 12:23:39.686333 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:39.686336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:39.686392 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:39.686496 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:39.687356 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.687881 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.688995 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.689290 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.689404 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.690522 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:39.690526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:39.690529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:39.690547 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:39.690565 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:39.690573 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:39.690660 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:39.690849 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 12:23:39.690911 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:23:39.691850 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.684162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:45033)
0620 12:23:39.684782 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:39.684820 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:39.686109 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:39.686660 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:39.686666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:39.686748 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:39.687209 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.687221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.688003 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.688008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.688868 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.688876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.689440 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.689444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.689616 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.690267 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:39.690269 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:39.690306 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:39.690364 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:39.690685 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:39.690692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:39.690697 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 12:23:39.690839 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 12:23:39.691095 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260620 12:23:39.692039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--builtin_ntp_servers=127.0.98.20:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 12:23:39.693141 20442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260620 12:23:39.693394 20442 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:39.693881 20442 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260620 12:23:39.694957 20123 ts_manager.cc:194] Registered new tserver with Master: d74d706eb2604a29bc060308e0fe2819 (127.0.98.2:41029)
I20260620 12:23:39.695792 20123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45033
W20260620 12:23:39.802387 20445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:39.802610 20445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:39.802667 20445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:39.806619 20445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:39.806668 20445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:39.806720 20445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 12:23:39.811664 20445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:39.812578 20445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:39.813920 20445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:39.823037 20450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:39.823056 20453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:39.823038 20451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:39.823235 20445 server_base.cc:1034] running on GCE node
I20260620 12:23:39.823652 20445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:39.824282 20445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:39.825453 20445 hybrid_clock.cc:648] HybridClock initialized: now 1781958219825434 us; error 38 us; skew 500 ppm
I20260620 12:23:39.827206 20445 webserver.cc:460] Webserver started at http://127.0.98.3:33815/ using document root <none> and password file <none>
I20260620 12:23:39.827755 20445 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:39.827845 20445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:39.828063 20445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:39.830058 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/data/instance:
uuid: "75c14545542e4e85bd3a3ffc87f524e4"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "866055f32687613497b4a2bdb0ac3c2e"
server_key_iv: "f2952b52c46cd4f33714dfad78b90d9e"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.830592 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75c14545542e4e85bd3a3ffc87f524e4"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "866055f32687613497b4a2bdb0ac3c2e"
server_key_iv: "f2952b52c46cd4f33714dfad78b90d9e"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.834055 20445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 12:23:39.836200 20459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.837421 20445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:39.837556 20445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/wal
uuid: "75c14545542e4e85bd3a3ffc87f524e4"
format_stamp: "Formatted at 2026-06-20 12:23:39 on dist-test-slave-9b3m"
server_key: "866055f32687613497b4a2bdb0ac3c2e"
server_key_iv: "f2952b52c46cd4f33714dfad78b90d9e"
server_key_version: "encryptionkey@0"
I20260620 12:23:39.837666 20445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:39.853468 20445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:39.854074 20445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:39.854254 20445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:39.854787 20445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 12:23:39.855665 20445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 12:23:39.855736 20445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.855810 20445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 12:23:39.855855 20445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:39.865023 20445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36699
I20260620 12:23:39.865103 20572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36699 every 8 connection(s)
I20260620 12:23:39.866017 20445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/data/info.pb
I20260620 12:23:39.868057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 20445
I20260620 12:23:39.868188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/wal/instance
I20260620 12:23:39.868460 392 external_mini_cluster.cc:1428] Setting key ac4a7fd90cad4b1ebd9e88979a861604
I20260620 12:23:39.876374 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.867813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44605 (local address 127.0.98.62:41805)
0620 12:23:39.868002 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:39.868007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:39.869277 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 12:23:39.871229 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:39.871232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:39.871289 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:39.871357 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:39.872321 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.872911 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.874370 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.874639 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.874755 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.875900 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:39.875904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:39.875906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:39.875920 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:39.875942 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:39.875952 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:39.876027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:39.876181 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 12:23:39.876237 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 12:23:39.877270 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.868298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:44605)
0620 12:23:39.869120 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:39.869174 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:39.870966 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:39.871518 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:39.871525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:39.871609 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:23:39.872128 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.872140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.873315 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.873319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.874224 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.874233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.874819 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.874823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.874977 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.875620 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 12:23:39.875621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:23:39.875659 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 12:23:39.875730 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:23:39.876041 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:23:39.876048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:23:39.876052 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 12:23:39.876149 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 12:23:39.876381 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":139,"threads_started":1}
I20260620 12:23:39.878474 20573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260620 12:23:39.878765 20573 heartbeater.cc:464] Registering TS with master...
I20260620 12:23:39.879281 20573 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260620 12:23:39.880285 20123 ts_manager.cc:194] Registered new tserver with Master: 75c14545542e4e85bd3a3ffc87f524e4 (127.0.98.3:36699)
I20260620 12:23:39.880941 20123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44605
I20260620 12:23:39.882395 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 12:23:39.888276 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.884281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.0.98.62:41805)
0620 12:23:39.884408 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:39.884414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:39.884600 (+ 186us) server_negotiation.cc:407] Connection header received
0620 12:23:39.884786 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:39.884789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:39.884836 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:39.884918 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:39.885258 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.885751 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.886590 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.886823 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.886931 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.887871 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:39.887874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:39.887876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:39.887887 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:39.887906 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:39.887913 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:39.887996 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:39.888093 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 12:23:39.888147 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:23:39.891021 20122 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 12:23:39.893941 20122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 12:23:39.908707 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.902314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36699 (local address 127.0.0.1:42928)
0620 12:23:39.903293 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:39.903347 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:39.903502 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:39.904183 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:39.904189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:39.904223 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:39.904422 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.904430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.905647 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.905653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.907004 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.907014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.908151 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.908155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.908266 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.908313 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 12:23:39.908412 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 12:23:39.908525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":43,"spinlock_wait_cycles":4864,"thread_start_us":110,"threads_started":1}
I20260620 12:23:39.908720 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.902482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.0.98.3:36699)
0620 12:23:39.902988 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:39.903001 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:39.903431 (+ 430us) server_negotiation.cc:407] Connection header received
0620 12:23:39.903693 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:39.903699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:39.903911 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:39.904014 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:39.904532 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.905482 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.907137 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.908011 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.908133 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.908353 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:39.908441 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 12:23:39.908541 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":187,"threads_started":1}
I20260620 12:23:39.908715 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.902314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41029 (local address 127.0.0.1:44316)
0620 12:23:39.903549 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:39.903568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:39.903656 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:39.904183 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:39.904189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:39.904225 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:39.904464 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.904473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.905952 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.905956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.907173 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.907185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.908271 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.908275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.908375 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.908411 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 12:23:39.908467 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 12:23:39.908540 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":103,"spinlock_wait_cycles":15232,"thread_start_us":67,"threads_started":1}
I20260620 12:23:39.908977 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.903050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41559 (local address 127.0.0.1:48388)
0620 12:23:39.903336 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:39.903353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:39.903485 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:39.904342 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:39.904345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:39.904364 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:39.904497 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.904506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.905952 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.905956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.907540 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:39.907547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.908624 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.908627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:39.908706 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.908737 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 12:23:39.908811 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 12:23:39.908889 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":28,"thread_start_us":79,"threads_started":1}
I20260620 12:23:39.909420 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.903272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.0.98.1:41559)
0620 12:23:39.903906 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:39.903919 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:39.903941 (+ 22us) server_negotiation.cc:407] Connection header received
0620 12:23:39.904015 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:39.904020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:39.904204 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:39.904351 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:39.904607 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.905821 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.907659 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.908275 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.908388 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.908563 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:39.909181 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 12:23:39.909289 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":118,"threads_started":1}
I20260620 12:23:39.909806 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:39.902700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.0.98.2:41029)
0620 12:23:39.903074 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:39.903086 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:39.903687 (+ 601us) server_negotiation.cc:407] Connection header received
0620 12:23:39.903783 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:39.903788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:39.903999 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:39.904156 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:39.904580 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.905814 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.907300 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:39.908119 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:39.909283 (+ 1164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:39.909463 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:39.909513 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 12:23:39.909611 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260620 12:23:39.912256 20245 tablet_service.cc:1467] Processing CreateTablet for tablet 6e89946908ea45dcbfc35c1cacff7740 (DEFAULT_TABLE table=test-table [id=76d7fe4f291d435984d815be786c87ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:39.912408 20507 tablet_service.cc:1467] Processing CreateTablet for tablet 6e89946908ea45dcbfc35c1cacff7740 (DEFAULT_TABLE table=test-table [id=76d7fe4f291d435984d815be786c87ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:39.912408 20376 tablet_service.cc:1467] Processing CreateTablet for tablet 6e89946908ea45dcbfc35c1cacff7740 (DEFAULT_TABLE table=test-table [id=76d7fe4f291d435984d815be786c87ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 12:23:39.913347 20245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e89946908ea45dcbfc35c1cacff7740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:39.913408 20376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e89946908ea45dcbfc35c1cacff7740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:39.913408 20507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e89946908ea45dcbfc35c1cacff7740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:39.919865 20590 tablet_bootstrap.cc:492] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b: Bootstrap starting.
I20260620 12:23:39.921109 20592 tablet_bootstrap.cc:492] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: Bootstrap starting.
I20260620 12:23:39.921363 20591 tablet_bootstrap.cc:492] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Bootstrap starting.
I20260620 12:23:39.922369 20590 tablet_bootstrap.cc:654] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:39.923177 20590 log.cc:824] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:39.923334 20591 tablet_bootstrap.cc:654] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:39.923334 20592 tablet_bootstrap.cc:654] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:39.924165 20591 log.cc:824] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:39.924171 20592 log.cc:824] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:39.925454 20590 tablet_bootstrap.cc:492] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b: No bootstrap required, opened a new log
I20260620 12:23:39.925673 20590 ts_tablet_manager.cc:1397] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 12:23:39.926128 20592 tablet_bootstrap.cc:492] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: No bootstrap required, opened a new log
I20260620 12:23:39.926128 20591 tablet_bootstrap.cc:492] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: No bootstrap required, opened a new log
I20260620 12:23:39.926311 20591 ts_tablet_manager.cc:1397] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 12:23:39.926311 20592 ts_tablet_manager.cc:1397] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 12:23:39.929234 20590 raft_consensus.cc:348] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:39.929478 20590 raft_consensus.cc:374] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:39.929543 20590 raft_consensus.cc:729] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b369fc8e13e4ed6b1d420642ba28f5b, State: Initialized, Role: FOLLOWER
I20260620 12:23:39.930033 20590 consensus_queue.cc:260] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:39.930709 20311 heartbeater.cc:503] Master 127.0.98.62:41805 was elected leader, sending a full tablet report...
I20260620 12:23:39.930547 20591 raft_consensus.cc:348] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:39.930784 20591 raft_consensus.cc:374] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:39.930841 20591 raft_consensus.cc:729] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c14545542e4e85bd3a3ffc87f524e4, State: Initialized, Role: FOLLOWER
I20260620 12:23:39.930547 20592 raft_consensus.cc:348] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:39.930954 20592 raft_consensus.cc:374] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:39.931000 20592 raft_consensus.cc:729] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74d706eb2604a29bc060308e0fe2819, State: Initialized, Role: FOLLOWER
I20260620 12:23:39.931077 20590 ts_tablet_manager.cc:1428] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 12:23:39.931361 20592 consensus_queue.cc:260] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:39.931361 20591 consensus_queue.cc:260] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:39.932266 20592 ts_tablet_manager.cc:1428] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:23:39.932266 20591 ts_tablet_manager.cc:1428] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 12:23:39.932591 20442 heartbeater.cc:503] Master 127.0.98.62:41805 was elected leader, sending a full tablet report...
W20260620 12:23:39.933007 20443 tablet.cc:2369] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:39.933825 20573 heartbeater.cc:503] Master 127.0.98.62:41805 was elected leader, sending a full tablet report...
W20260620 12:23:39.983652 20312 tablet.cc:2369] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:40.087954 20596 raft_consensus.cc:482] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 12:23:40.088196 20596 raft_consensus.cc:504] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:40.089607 20596 leader_election.cc:288] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c14545542e4e85bd3a3ffc87f524e4 (127.0.98.3:36699), d74d706eb2604a29bc060308e0fe2819 (127.0.98.2:41029)
I20260620 12:23:40.093374 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:40.089998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35381 (local address 127.0.98.3:36699)
0620 12:23:40.090133 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:40.090141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:40.090188 (+ 47us) server_negotiation.cc:407] Connection header received
0620 12:23:40.090403 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:40.090407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:40.090456 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:40.090543 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:40.090818 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.091311 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.092351 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.092756 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.092875 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:40.092936 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:40.093179 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 12:23:40.093252 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 12:23:40.093396 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:40.089819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36699 (local address 127.0.98.1:35381)
0620 12:23:40.090104 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:40.090120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:40.090293 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:40.090557 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:40.090559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:40.090576 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:40.090702 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:40.090707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.091428 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.091434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:40.092214 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:40.092232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.092878 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.092881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:40.092979 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:40.093010 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 12:23:40.093159 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 12:23:40.093239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260620 12:23:40.093910 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:40.089990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41029 (local address 127.0.98.1:48963)
0620 12:23:40.090362 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:40.090378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:23:40.090470 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:23:40.090778 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:23:40.090781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:23:40.090798 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 12:23:40.090941 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:40.090946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.091748 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.091755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:40.092632 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:23:40.092641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.093415 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.093419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:23:40.093533 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:40.093556 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 12:23:40.093664 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:23:40.093740 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":23,"thread_start_us":121,"threads_started":1}
I20260620 12:23:40.093930 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:40.090248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48963 (local address 127.0.98.2:41029)
0620 12:23:40.090400 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:40.090408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:40.090441 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:23:40.090602 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:40.090606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:40.090664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:40.090765 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 12:23:40.091069 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.091623 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.092766 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.093286 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.093423 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:40.093485 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:40.093719 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 12:23:40.093799 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:23:40.094214 20527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e89946908ea45dcbfc35c1cacff7740" candidate_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c14545542e4e85bd3a3ffc87f524e4" is_pre_election: true
I20260620 12:23:40.094553 20527 raft_consensus.cc:2454] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b369fc8e13e4ed6b1d420642ba28f5b in term 0.
I20260620 12:23:40.094651 20396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e89946908ea45dcbfc35c1cacff7740" candidate_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74d706eb2604a29bc060308e0fe2819" is_pre_election: true
I20260620 12:23:40.094959 20396 raft_consensus.cc:2454] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b369fc8e13e4ed6b1d420642ba28f5b in term 0.
I20260620 12:23:40.095048 20201 leader_election.cc:302] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b369fc8e13e4ed6b1d420642ba28f5b, 75c14545542e4e85bd3a3ffc87f524e4; no voters:
I20260620 12:23:40.095400 20596 raft_consensus.cc:2781] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 12:23:40.095474 20596 raft_consensus.cc:482] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 12:23:40.095503 20596 raft_consensus.cc:3037] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:40.096527 20596 raft_consensus.cc:504] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:40.096864 20596 leader_election.cc:288] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [CANDIDATE]: Term 1 election: Requested vote from peers 75c14545542e4e85bd3a3ffc87f524e4 (127.0.98.3:36699), d74d706eb2604a29bc060308e0fe2819 (127.0.98.2:41029)
I20260620 12:23:40.097256 20527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e89946908ea45dcbfc35c1cacff7740" candidate_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c14545542e4e85bd3a3ffc87f524e4"
I20260620 12:23:40.097332 20396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e89946908ea45dcbfc35c1cacff7740" candidate_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74d706eb2604a29bc060308e0fe2819"
I20260620 12:23:40.097407 20527 raft_consensus.cc:3037] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:40.097451 20396 raft_consensus.cc:3037] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:40.098428 20396 raft_consensus.cc:2454] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b369fc8e13e4ed6b1d420642ba28f5b in term 1.
I20260620 12:23:40.098526 20527 raft_consensus.cc:2454] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b369fc8e13e4ed6b1d420642ba28f5b in term 1.
I20260620 12:23:40.098737 20198 leader_election.cc:302] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b369fc8e13e4ed6b1d420642ba28f5b, d74d706eb2604a29bc060308e0fe2819; no voters:
I20260620 12:23:40.098989 20596 raft_consensus.cc:2781] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:40.099212 20596 raft_consensus.cc:686] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 1 LEADER]: Becoming Leader. State: Replica: 6b369fc8e13e4ed6b1d420642ba28f5b, State: Running, Role: LEADER
I20260620 12:23:40.099546 20596 consensus_queue.cc:237] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } }
I20260620 12:23:40.102581 20122 catalog_manager.cc:5495] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b369fc8e13e4ed6b1d420642ba28f5b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b369fc8e13e4ed6b1d420642ba28f5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41559 } health_report { overall_health: HEALTHY } } }
W20260620 12:23:40.116987 20574 tablet.cc:2369] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:23:40.144250 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:40.140599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.0.98.1:41559)
0620 12:23:40.140755 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:40.140759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:40.140774 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:23:40.140900 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:40.140902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:40.140958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:40.141099 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:23:40.141344 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.141920 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.142788 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:40.143032 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:40.143142 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:40.143449 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:23:40.143864 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:23:40.143999 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:40.144058 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 12:23:40.144108 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 12:23:40.150184 20527 raft_consensus.cc:1261] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 1 FOLLOWER]: Refusing update from remote peer 6b369fc8e13e4ed6b1d420642ba28f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:40.150184 20396 raft_consensus.cc:1261] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 1 FOLLOWER]: Refusing update from remote peer 6b369fc8e13e4ed6b1d420642ba28f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:23:40.150928 20596 consensus_queue.cc:1035] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c14545542e4e85bd3a3ffc87f524e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:40.151105 20601 consensus_queue.cc:1035] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74d706eb2604a29bc060308e0fe2819" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:23:40.157912 20604 mvcc.cc:205] Tried to move back new op lower bound from 7298900869727952896 to 7298900869530468352. Current Snapshot: MvccSnapshot[applied={T|T < 7298900869727952896}]
I20260620 12:23:40.158753 20605 mvcc.cc:205] Tried to move back new op lower bound from 7298900869727952896 to 7298900869530468352. Current Snapshot: MvccSnapshot[applied={T|T < 7298900869727952896}]
I20260620 12:23:40.165202 20606 mvcc.cc:205] Tried to move back new op lower bound from 7298900869727952896 to 7298900869530468352. Current Snapshot: MvccSnapshot[applied={T|T < 7298900869727952896 or (T in {7298900869727952896})}]
I20260620 12:23:40.166811 20122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:40.166957 20122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 12:23:40.169461 20122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11b6e6d5e86e457b9c3e9bbfdc45bb5e: Sending DeleteTablet for 3 replicas of tablet 6e89946908ea45dcbfc35c1cacff7740
I20260620 12:23:40.170264 20376 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e89946908ea45dcbfc35c1cacff7740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:40 UTC) from {username='slave'} at 127.0.0.1:44316
I20260620 12:23:40.170248 20507 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e89946908ea45dcbfc35c1cacff7740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:40 UTC) from {username='slave'} at 127.0.0.1:42928
I20260620 12:23:40.170758 20614 tablet_replica.cc:330] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: stopping tablet replica
I20260620 12:23:40.170907 20245 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e89946908ea45dcbfc35c1cacff7740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 12:23:40 UTC) from {username='slave'} at 127.0.0.1:48388
I20260620 12:23:40.171176 20614 raft_consensus.cc:2229] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:40.171409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 20183
I20260620 12:23:40.171521 20615 tablet_replica.cc:330] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b: stopping tablet replica
I20260620 12:23:40.171561 20614 raft_consensus.cc:2258] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:40.171825 20615 raft_consensus.cc:2229] T 6e89946908ea45dcbfc35c1cacff7740 P 6b369fc8e13e4ed6b1d420642ba28f5b [term 1 LEADER]: Raft consensus shutting down.
I20260620 12:23:40.172158 20613 tablet_replica.cc:330] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: stopping tablet replica
I20260620 12:23:40.172456 20613 raft_consensus.cc:2229] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 12:23:40.172741 20613 raft_consensus.cc:2258] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 12:23:40.173532 20614 ts_tablet_manager.cc:1905] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:40.176978 20614 ts_tablet_manager.cc:1918] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 12:23:40.177098 20614 log.cc:1196] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Deleting WAL directory at /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781957967886955-392-0/minicluster-data/ts-2/wal/wals/6e89946908ea45dcbfc35c1cacff7740
I20260620 12:23:40.177417 20614 ts_tablet_manager.cc:1939] T 6e89946908ea45dcbfc35c1cacff7740 P 75c14545542e4e85bd3a3ffc87f524e4: Deleting consensus metadata
I20260620 12:23:40.177892 20613 ts_tablet_manager.cc:1905] T 6e89946908ea45dcbfc35c1cacff7740 P d74d706eb2604a29bc060308e0fe2819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 12:23:40.178651 20110 catalog_manager.cc:4841] TS 75c14545542e4e85bd3a3ffc87f524e4 (127.0.98.3:36699): tablet 6e89946908ea45dcbfc35c1cacff7740 (table test-table [id=76d7fe4f291d435984d815be786c87ce]) successfully deleted
W20260620 12:23:40.179921 20107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41559 (error 108)
I20260620 12:23:40.179975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 20314
I20260620 12:23:40.182221 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:40.181980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41559 (local address 127.0.0.1:48404)
0620 12:23:40.182100 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:40.182155 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 12:23:40.182466 20107 catalog_manager.cc:4584] TS 6b369fc8e13e4ed6b1d420642ba28f5b (127.0.98.1:41559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e89946908ea45dcbfc35c1cacff7740: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41559: connect: Connection refused (error 111)
I20260620 12:23:40.185652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 20445
I20260620 12:23:40.186666 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:40.185844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41029 (local address 127.0.0.1:44322)
0620 12:23:40.186522 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 12:23:40.186583 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590}
I20260620 12:23:40.191063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 20092
2026-06-20T12:23:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1145 ms)
[----------] 6 tests from EncryptionPolicyTest (10857 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:40 dist-test-slave-9b3m krb5kdc[20619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:40 dist-test-slave-9b3m krb5kdc[20619](info): set up 2 sockets
Jun 20 12:23:40 dist-test-slave-9b3m krb5kdc[20619](info): commencing operation
krb5kdc: starting...
W20260620 12:23:41.233202 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:41 dist-test-slave-9b3m krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:41.356459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:41.466892 20632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:41.467212 20632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:41.467294 20632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:41.471292 20632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:41.471390 20632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:41.471429 20632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:41.471489 20632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:41.471527 20632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:41.476835 20632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:41.477945 20632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:41.478974 20632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:41.487365 20638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:41.487408 20637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:41.487365 20640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:41.487735 20632 server_base.cc:1034] running on GCE node
I20260620 12:23:41.488289 20632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:41.489418 20632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:41.490628 20632 hybrid_clock.cc:648] HybridClock initialized: now 1781958221490617 us; error 33 us; skew 500 ppm
Jun 20 12:23:41 dist-test-slave-9b3m krb5kdc[20619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:41.493472 20632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:41.494347 20632 webserver.cc:460] Webserver started at http://127.0.0.1:38015/ using document root <none> and password file <none>
I20260620 12:23:41.494901 20632 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:41.494995 20632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:41.495224 20632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:41.497318 20632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "f89ee9cf6eed444f984eb8f168f3e873"
format_stamp: "Formatted at 2026-06-20 12:23:41 on dist-test-slave-9b3m"
server_key: "44b6ec027722859e2fba19f6b55b818b"
server_key_iv: "176061dcbea00650c85c1a622c166b62"
server_key_version: "encryptionkey@0"
I20260620 12:23:41.497884 20632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "f89ee9cf6eed444f984eb8f168f3e873"
format_stamp: "Formatted at 2026-06-20 12:23:41 on dist-test-slave-9b3m"
server_key: "44b6ec027722859e2fba19f6b55b818b"
server_key_iv: "176061dcbea00650c85c1a622c166b62"
server_key_version: "encryptionkey@0"
I20260620 12:23:41.501469 20632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:23:41.503789 20647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:41.505188 20632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:41.505339 20632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "f89ee9cf6eed444f984eb8f168f3e873"
format_stamp: "Formatted at 2026-06-20 12:23:41 on dist-test-slave-9b3m"
server_key: "44b6ec027722859e2fba19f6b55b818b"
server_key_iv: "176061dcbea00650c85c1a622c166b62"
server_key_version: "encryptionkey@0"
I20260620 12:23:41.505470 20632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:41.521013 20632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:41.524158 20632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:41.524363 20632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:41.531378 20632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35195
I20260620 12:23:41.531399 20699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35195 every 8 connection(s)
I20260620 12:23:41.532295 20632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:41.534997 20700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:41.540313 20700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873: Bootstrap starting.
I20260620 12:23:41.542502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 20632
I20260620 12:23:41.542644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:41.542817 20700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:41.542969 392 external_mini_cluster.cc:1428] Setting key 6e9cc6285d08afb4059033dc9f71aba1
I20260620 12:23:41.543637 20700 log.cc:824] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:41.545822 20700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873: No bootstrap required, opened a new log
I20260620 12:23:41.550189 20700 raft_consensus.cc:348] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89ee9cf6eed444f984eb8f168f3e873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } }
I20260620 12:23:41.550486 20700 raft_consensus.cc:374] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:41.550559 20700 raft_consensus.cc:729] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89ee9cf6eed444f984eb8f168f3e873, State: Initialized, Role: FOLLOWER
I20260620 12:23:41.550977 20700 consensus_queue.cc:260] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89ee9cf6eed444f984eb8f168f3e873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } }
I20260620 12:23:41.551137 20700 raft_consensus.cc:388] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:41.551224 20700 raft_consensus.cc:482] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:41.551345 20700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:41.552381 20700 raft_consensus.cc:504] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89ee9cf6eed444f984eb8f168f3e873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } }
Jun 20 12:23:41 dist-test-slave-9b3m krb5kdc[20619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:23:41.552728 20700 leader_election.cc:302] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f89ee9cf6eed444f984eb8f168f3e873; no voters:
I20260620 12:23:41.553093 20700 leader_election.cc:288] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:41.553208 20705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:41.553460 20705 raft_consensus.cc:686] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [term 1 LEADER]: Becoming Leader. State: Replica: f89ee9cf6eed444f984eb8f168f3e873, State: Running, Role: LEADER
I20260620 12:23:41.554116 20705 consensus_queue.cc:237] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89ee9cf6eed444f984eb8f168f3e873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } }
I20260620 12:23:41.554433 20700 sys_catalog.cc:564] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:41.555749 20707 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f89ee9cf6eed444f984eb8f168f3e873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89ee9cf6eed444f984eb8f168f3e873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } } }
I20260620 12:23:41.555886 20707 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:41.556258 20706 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f89ee9cf6eed444f984eb8f168f3e873. Latest consensus state: current_term: 1 leader_uuid: "f89ee9cf6eed444f984eb8f168f3e873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89ee9cf6eed444f984eb8f168f3e873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } } }
I20260620 12:23:41.556408 20706 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:41.557221 20716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:41.558710 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:41.544175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50163 (local address 127.0.0.1:35195)
0620 12:23:41.544594 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:41.544613 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:41.544653 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:23:41.545461 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:41.545484 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:41.545906 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:41.546259 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:41.546925 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:41.548415 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:41.549356 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:41.550052 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:41.550220 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:41.553150 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:41.553157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:41.553172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:41.553201 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:41.555282 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:41.555937 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:41.555945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:41.555954 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:41.556107 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:41.556578 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:41.556584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:41.556589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:41.557166 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:41.557399 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:41.557708 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 12:23:41.557947 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260620 12:23:41.560467 20716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:41.561566 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:41.566836 20716 catalog_manager.cc:1347] Generated new cluster ID: f727188574d241cba3fa4e647efee810
I20260620 12:23:41.566905 20716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:41.572489 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:41.563566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38649 (local address 127.0.0.1:35195)
0620 12:23:41.563673 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:41.563677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:41.563872 (+ 195us) server_negotiation.cc:407] Connection header received
0620 12:23:41.563969 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:41.563972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:41.564020 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:41.564107 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:41.564876 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:41.565972 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:41.566944 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:41.567287 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:41.567362 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:41.569127 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:41.569131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:41.569134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:41.569157 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:41.570708 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:41.571217 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:41.571220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:41.571222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:41.571290 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:41.571595 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:41.571599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:41.571600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:41.571777 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:41.571876 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:41.572253 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 12:23:41.572343 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 12:23:41.574894 20716 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:41.575736 20716 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:41.584849 20716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f89ee9cf6eed444f984eb8f168f3e873: Generated new TSK 0
I20260620 12:23:41.585568 20716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:41.591687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 20632
W20260620 12:23:41.597008 20617 connection.cc:535] client connection to 127.0.0.1:35195 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35195): Connection reset by peer (error 104)
2026-06-20T12:23:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:41 dist-test-slave-9b3m krb5kdc[20731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:41 dist-test-slave-9b3m krb5kdc[20731](info): set up 2 sockets
Jun 20 12:23:41 dist-test-slave-9b3m krb5kdc[20731](info): commencing operation
krb5kdc: starting...
W20260620 12:23:42.721298 392 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:42 dist-test-slave-9b3m krb5kdc[20731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:42.840178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:42.947503 20744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:42.947788 20744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:42.947876 20744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:42.951826 20744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:42.951912 20744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:42.951951 20744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:42.951999 20744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:42.952037 20744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:42.957193 20744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:42.958253 20744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:42.959254 20744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:42.967577 20752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:42.967577 20750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:42.967666 20744 server_base.cc:1034] running on GCE node
W20260620 12:23:42.967585 20749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:42.968425 20744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:42.969530 20744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:42.970714 20744 hybrid_clock.cc:648] HybridClock initialized: now 1781958222970700 us; error 26 us; skew 500 ppm
Jun 20 12:23:42 dist-test-slave-9b3m krb5kdc[20731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:42.973349 20744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:42.974226 20744 webserver.cc:460] Webserver started at http://127.0.0.1:39193/ using document root <none> and password file <none>
I20260620 12:23:42.974743 20744 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:42.974797 20744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:42.974985 20744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:42.976955 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "b66297a2ac61459faaa5610514ce7821"
format_stamp: "Formatted at 2026-06-20 12:23:42 on dist-test-slave-9b3m"
server_key: "d5bd038f5a015cb29f80d5fc3e1fa79f"
server_key_iv: "96a7925612db725732649005f96ca6af"
server_key_version: "encryptionkey@0"
I20260620 12:23:42.977509 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "b66297a2ac61459faaa5610514ce7821"
format_stamp: "Formatted at 2026-06-20 12:23:42 on dist-test-slave-9b3m"
server_key: "d5bd038f5a015cb29f80d5fc3e1fa79f"
server_key_iv: "96a7925612db725732649005f96ca6af"
server_key_version: "encryptionkey@0"
I20260620 12:23:42.980892 20744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:23:42.983198 20759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:42.984494 20744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:42.984611 20744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "b66297a2ac61459faaa5610514ce7821"
format_stamp: "Formatted at 2026-06-20 12:23:42 on dist-test-slave-9b3m"
server_key: "d5bd038f5a015cb29f80d5fc3e1fa79f"
server_key_iv: "96a7925612db725732649005f96ca6af"
server_key_version: "encryptionkey@0"
I20260620 12:23:42.984715 20744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:43.013823 20744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:43.017136 20744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:43.017289 20744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:43.024236 20744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46533
I20260620 12:23:43.024267 20811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46533 every 8 connection(s)
I20260620 12:23:43.025207 20744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:43.025945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 20744
I20260620 12:23:43.026098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:43.026398 392 external_mini_cluster.cc:1428] Setting key ff9729a5702b7698b5aaffd614358db5
I20260620 12:23:43.028497 20812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:43.034802 20812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821: Bootstrap starting.
I20260620 12:23:43.035188 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:43.027697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34629 (local address 127.0.0.1:46533)
0620 12:23:43.028084 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:43.028099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:43.028137 (+ 38us) server_negotiation.cc:407] Connection header received
0620 12:23:43.028830 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:43.028848 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:43.029198 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:43.029465 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:43.029771 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:43.030727 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:43.031930 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:43.032368 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:43.032478 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:43.033482 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:43.033487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:43.033499 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:43.033878 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:43.033929 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:43.033951 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:43.034048 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:43.034289 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 12:23:43.034428 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
I20260620 12:23:43.037305 20812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:43.038115 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:43.038209 20812 log.cc:824] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:43.040395 20812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821: No bootstrap required, opened a new log
I20260620 12:23:43.044379 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:43.040257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34593 (local address 127.0.0.1:46533)
0620 12:23:43.040366 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:43.040375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:43.040513 (+ 138us) server_negotiation.cc:407] Connection header received
0620 12:23:43.040676 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:43.040678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:43.040713 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:43.040776 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:43.041094 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:43.041573 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:43.042806 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:43.043001 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:43.043074 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:43.043880 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:43.043882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:43.043884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:43.043894 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:43.043917 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:43.043927 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:43.043993 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:43.044173 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 12:23:43.044228 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 12:23:43.044976 20812 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66297a2ac61459faaa5610514ce7821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } }
I20260620 12:23:43.045270 20812 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:43.045336 20812 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66297a2ac61459faaa5610514ce7821, State: Initialized, Role: FOLLOWER
I20260620 12:23:43.045835 20812 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66297a2ac61459faaa5610514ce7821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } }
I20260620 12:23:43.046053 20812 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:43.046128 20812 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:43.046233 20812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:43.047436 20812 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66297a2ac61459faaa5610514ce7821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } }
I20260620 12:23:43.047822 20812 leader_election.cc:302] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66297a2ac61459faaa5610514ce7821; no voters:
I20260620 12:23:43.048255 20812 leader_election.cc:288] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:43.048341 20822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:43.048513 20822 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [term 1 LEADER]: Becoming Leader. State: Replica: b66297a2ac61459faaa5610514ce7821, State: Running, Role: LEADER
I20260620 12:23:43.048784 20822 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66297a2ac61459faaa5610514ce7821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } }
I20260620 12:23:43.049527 20812 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:43.049927 20822 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66297a2ac61459faaa5610514ce7821. Latest consensus state: current_term: 1 leader_uuid: "b66297a2ac61459faaa5610514ce7821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66297a2ac61459faaa5610514ce7821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } } }
I20260620 12:23:43.050031 20822 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:43.050794 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66297a2ac61459faaa5610514ce7821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66297a2ac61459faaa5610514ce7821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46533 } } }
I20260620 12:23:43.050886 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:43.051740 20831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:43.055742 20831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:43.061117 20831 catalog_manager.cc:1347] Generated new cluster ID: b59139aa784b46478f31c16837e744ae
I20260620 12:23:43.061177 20831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:43.079339 20831 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:43.080583 20831 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:43.087546 20831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66297a2ac61459faaa5610514ce7821: Generated new TSK 0
I20260620 12:23:43.088338 20831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:43.093068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 20744
W20260620 12:23:43.098356 20729 connection.cc:535] client connection to 127.0.0.1:46533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46533): Connection reset by peer (error 104)
2026-06-20T12:23:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:43 dist-test-slave-9b3m krb5kdc[20843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:43 dist-test-slave-9b3m krb5kdc[20843](info): set up 2 sockets
Jun 20 12:23:43 dist-test-slave-9b3m krb5kdc[20843](info): commencing operation
krb5kdc: starting...
W20260620 12:23:44.129289 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:44 dist-test-slave-9b3m krb5kdc[20843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:44.248013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:44.360082 20856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:44.360585 20856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:44.360680 20856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:44.367671 20856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:44.367728 20856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:44.367748 20856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:44.367765 20856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:44.367780 20856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:44.376998 20856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:44.378129 20856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:44.379487 20856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:44.389868 20861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:44.389837 20864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:44.390142 20862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:44.390259 20856 server_base.cc:1034] running on GCE node
I20260620 12:23:44.390944 20856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:44.392478 20856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:44.393684 20856 hybrid_clock.cc:648] HybridClock initialized: now 1781958224393658 us; error 35 us; skew 500 ppm
Jun 20 12:23:44 dist-test-slave-9b3m krb5kdc[20843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:44.397517 20856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:44.398702 20856 webserver.cc:460] Webserver started at http://127.0.0.1:41973/ using document root <none> and password file <none>
I20260620 12:23:44.399399 20856 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:44.399464 20856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:44.399622 20856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:44.401679 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "0c28c08c77eb49da9111ea092825a8bb"
format_stamp: "Formatted at 2026-06-20 12:23:44 on dist-test-slave-9b3m"
server_key: "9d5b659c3dc4bcedd384299e2b6de1d7"
server_key_iv: "6e283e966df31c6f46bb5327d079400c"
server_key_version: "encryptionkey@0"
I20260620 12:23:44.402231 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c28c08c77eb49da9111ea092825a8bb"
format_stamp: "Formatted at 2026-06-20 12:23:44 on dist-test-slave-9b3m"
server_key: "9d5b659c3dc4bcedd384299e2b6de1d7"
server_key_iv: "6e283e966df31c6f46bb5327d079400c"
server_key_version: "encryptionkey@0"
I20260620 12:23:44.405738 20856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:23:44.407992 20871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:44.409307 20856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:44.409404 20856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "0c28c08c77eb49da9111ea092825a8bb"
format_stamp: "Formatted at 2026-06-20 12:23:44 on dist-test-slave-9b3m"
server_key: "9d5b659c3dc4bcedd384299e2b6de1d7"
server_key_iv: "6e283e966df31c6f46bb5327d079400c"
server_key_version: "encryptionkey@0"
I20260620 12:23:44.409487 20856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:44.413839 20856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:44.416596 20856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:44.416728 20856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:44.423759 20856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43369
I20260620 12:23:44.423792 20923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43369 every 8 connection(s)
I20260620 12:23:44.424597 20856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:44.427142 20924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:44.432260 20924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb: Bootstrap starting.
I20260620 12:23:44.433707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 20856
I20260620 12:23:44.433880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:44.434181 392 external_mini_cluster.cc:1428] Setting key b7714fb617ee96c7f9ae03b40147cbfd
I20260620 12:23:44.434958 20924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:44.435789 20924 log.cc:824] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:44.437911 20924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb: No bootstrap required, opened a new log
I20260620 12:23:44.439199 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:44.435207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38873 (local address 127.0.0.1:43369)
0620 12:23:44.435702 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:44.435719 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:44.435760 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:23:44.436507 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:44.436531 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:44.436870 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:44.437243 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:44.437908 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:44.437914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:44.437930 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:44.437945 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:44.438008 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:44.438034 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:44.438144 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:44.438213 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 12:23:44.438359 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":138,"threads_started":1}
I20260620 12:23:44.441781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:44.441763 20924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c28c08c77eb49da9111ea092825a8bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43369 } }
I20260620 12:23:44.441996 20924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:44.442070 20924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c28c08c77eb49da9111ea092825a8bb, State: Initialized, Role: FOLLOWER
I20260620 12:23:44.442469 20924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c28c08c77eb49da9111ea092825a8bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43369 } }
I20260620 12:23:44.442581 20924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:44.442668 20924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:44.442782 20924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:44.443840 20924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c28c08c77eb49da9111ea092825a8bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43369 } }
I20260620 12:23:44.444240 20924 leader_election.cc:302] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c28c08c77eb49da9111ea092825a8bb; no voters:
I20260620 12:23:44.444600 20924 leader_election.cc:288] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:44.444713 20934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:44.444918 20934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [term 1 LEADER]: Becoming Leader. State: Replica: 0c28c08c77eb49da9111ea092825a8bb, State: Running, Role: LEADER
I20260620 12:23:44.445273 20934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c28c08c77eb49da9111ea092825a8bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43369 } }
I20260620 12:23:44.445881 20924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:44.445897 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:44.443925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43561 (local address 127.0.0.1:43369)
0620 12:23:44.444078 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:44.444081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:44.444248 (+ 167us) server_negotiation.cc:407] Connection header received
0620 12:23:44.444416 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:44.444420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:44.444450 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:44.444527 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:44.445280 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:44.445287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:44.445293 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:44.445304 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:44.445330 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:44.445342 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:44.445427 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:44.445755 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 12:23:44.445792 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 12:23:44.447546 20935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c28c08c77eb49da9111ea092825a8bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c28c08c77eb49da9111ea092825a8bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43369 } } }
I20260620 12:23:44.447683 20935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:44.448046 20936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c28c08c77eb49da9111ea092825a8bb. Latest consensus state: current_term: 1 leader_uuid: "0c28c08c77eb49da9111ea092825a8bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c28c08c77eb49da9111ea092825a8bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43369 } } }
I20260620 12:23:44.448161 20936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:44.448112 20948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:44.450839 20948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:44.455761 20948 catalog_manager.cc:1347] Generated new cluster ID: 07ab4bfa07ca46c09f0859dc631b86aa
I20260620 12:23:44.455825 20948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:44.484951 20948 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:44.485299 20948 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:44.494184 20948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c28c08c77eb49da9111ea092825a8bb: Generated new TSK 0
I20260620 12:23:44.494828 20948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:44.503185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 20856
2026-06-20T12:23:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:44 dist-test-slave-9b3m krb5kdc[20955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:44 dist-test-slave-9b3m krb5kdc[20955](info): set up 2 sockets
Jun 20 12:23:44 dist-test-slave-9b3m krb5kdc[20955](info): commencing operation
krb5kdc: starting...
W20260620 12:23:45.537298 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:45 dist-test-slave-9b3m krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:45.653942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:45.761178 20968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:45.761473 20968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:45.761557 20968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:45.765538 20968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:45.765612 20968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:45.765650 20968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:45.765705 20968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:45.765738 20968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:45.770864 20968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:45.771870 20968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:45.772864 20968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:45.781445 20973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:45.781421 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:45.781426 20976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:45.781742 20968 server_base.cc:1034] running on GCE node
I20260620 12:23:45.782402 20968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:45.783416 20968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:45.784603 20968 hybrid_clock.cc:648] HybridClock initialized: now 1781958225784574 us; error 44 us; skew 500 ppm
Jun 20 12:23:45 dist-test-slave-9b3m krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:45.790930 20968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:45.791800 20968 webserver.cc:460] Webserver started at http://127.0.0.1:38497/ using document root <none> and password file <none>
I20260620 12:23:45.792295 20968 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:45.792361 20968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:45.792514 20968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:45.794479 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "25f547c5db294d968425a4a9cec51e45"
format_stamp: "Formatted at 2026-06-20 12:23:45 on dist-test-slave-9b3m"
server_key: "4cbaa5066643be6e7ae687cd8b8e4205"
server_key_iv: "d7d2eaabb67d88c6dcadd8b50bd5b4e6"
server_key_version: "encryptionkey@0"
I20260620 12:23:45.794965 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "25f547c5db294d968425a4a9cec51e45"
format_stamp: "Formatted at 2026-06-20 12:23:45 on dist-test-slave-9b3m"
server_key: "4cbaa5066643be6e7ae687cd8b8e4205"
server_key_iv: "d7d2eaabb67d88c6dcadd8b50bd5b4e6"
server_key_version: "encryptionkey@0"
I20260620 12:23:45.798465 20968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:45.800724 20983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:45.801931 20968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:45.802027 20968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "25f547c5db294d968425a4a9cec51e45"
format_stamp: "Formatted at 2026-06-20 12:23:45 on dist-test-slave-9b3m"
server_key: "4cbaa5066643be6e7ae687cd8b8e4205"
server_key_iv: "d7d2eaabb67d88c6dcadd8b50bd5b4e6"
server_key_version: "encryptionkey@0"
I20260620 12:23:45.802109 20968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:45.818342 20968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:45.821645 20968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:45.821852 20968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:45.828696 20968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40909
I20260620 12:23:45.828711 21035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40909 every 8 connection(s)
I20260620 12:23:45.829635 20968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:45.832484 21036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:45.838266 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45: Bootstrap starting.
I20260620 12:23:45.839469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 20968
I20260620 12:23:45.839623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:45.839916 392 external_mini_cluster.cc:1428] Setting key 66908f2c4c69944450ccade7a1a4682f
I20260620 12:23:45.841216 21036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:45.841991 21036 log.cc:824] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:45.844152 21036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45: No bootstrap required, opened a new log
I20260620 12:23:45.848137 21036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f547c5db294d968425a4a9cec51e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } }
I20260620 12:23:45.848371 21036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:45.848453 21036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f547c5db294d968425a4a9cec51e45, State: Initialized, Role: FOLLOWER
I20260620 12:23:45.848868 21036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f547c5db294d968425a4a9cec51e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } }
Jun 20 12:23:45 dist-test-slave-9b3m krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:23:45.849014 21036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:45.849126 21036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:45.849226 21036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:45.850268 21036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f547c5db294d968425a4a9cec51e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } }
I20260620 12:23:45.850627 21036 leader_election.cc:302] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25f547c5db294d968425a4a9cec51e45; no voters:
I20260620 12:23:45.850963 21036 leader_election.cc:288] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:45.851090 21041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:45.851320 21041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [term 1 LEADER]: Becoming Leader. State: Replica: 25f547c5db294d968425a4a9cec51e45, State: Running, Role: LEADER
I20260620 12:23:45.851647 21041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f547c5db294d968425a4a9cec51e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } }
I20260620 12:23:45.852150 21036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:45.853832 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f547c5db294d968425a4a9cec51e45. Latest consensus state: current_term: 1 leader_uuid: "25f547c5db294d968425a4a9cec51e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f547c5db294d968425a4a9cec51e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } } }
I20260620 12:23:45.853989 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:45.853832 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25f547c5db294d968425a4a9cec51e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f547c5db294d968425a4a9cec51e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40909 } } }
I20260620 12:23:45.854113 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:45.854596 21055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:45.855906 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:45.841014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33325 (local address 127.0.0.1:40909)
0620 12:23:45.841473 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:45.841488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:45.841522 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:23:45.842149 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:45.842172 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:45.842508 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:45.842789 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:45.843556 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:45.845022 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:45.845975 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:45.846697 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:45.846897 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:45.849645 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:45.849650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:45.849662 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:45.849692 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:45.852009 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:45.852507 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:45.852513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:45.852519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:45.852664 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:45.853037 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:45.853068 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:45.853070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:45.853572 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:45.853779 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:45.854728 (+ 949us) server_negotiation.cc:299] Negotiation successful
0620 12:23:45.855004 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260620 12:23:45.857630 21055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:45.859182 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:45.863656 21055 catalog_manager.cc:1347] Generated new cluster ID: 977d4b2d8bd64493994fe886ee8ad932
I20260620 12:23:45.863730 21055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:45.869372 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:45.861133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59027 (local address 127.0.0.1:40909)
0620 12:23:45.861235 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:45.861238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:45.861520 (+ 282us) server_negotiation.cc:407] Connection header received
0620 12:23:45.861656 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:45.861659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:45.861706 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:45.861788 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:45.862469 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:45.863556 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:45.864400 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:45.864642 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:45.864750 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:45.866103 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:45.866107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:45.866110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:45.866133 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:45.867558 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:45.868000 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:45.868005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:45.868007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:45.868089 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:45.868422 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:45.868427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:45.868428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:45.868595 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:45.868740 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:45.868985 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 12:23:45.869120 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 12:23:45.891100 21055 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:45.892314 21055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:45.899080 21055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25f547c5db294d968425a4a9cec51e45: Generated new TSK 0
I20260620 12:23:45.899830 21055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:45.912099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 20968
2026-06-20T12:23:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:45 dist-test-slave-9b3m krb5kdc[21067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:45 dist-test-slave-9b3m krb5kdc[21067](info): set up 2 sockets
Jun 20 12:23:45 dist-test-slave-9b3m krb5kdc[21067](info): commencing operation
krb5kdc: starting...
W20260620 12:23:46.969274 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:47 dist-test-slave-9b3m krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:47.088366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:47.194761 21080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:47.195062 21080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:47.195138 21080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:47.199088 21080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:47.199187 21080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:47.199226 21080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:47.199278 21080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:47.199317 21080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:47.205094 21080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:47.206137 21080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:47.207144 21080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:47.215539 21088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:47.215557 21086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:47.215530 21085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:47.215955 21080 server_base.cc:1034] running on GCE node
I20260620 12:23:47.216485 21080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:47.217669 21080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:47.218875 21080 hybrid_clock.cc:648] HybridClock initialized: now 1781958227218860 us; error 25 us; skew 500 ppm
Jun 20 12:23:47 dist-test-slave-9b3m krb5kdc[21067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:47.221695 21080 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:47.222527 21080 webserver.cc:460] Webserver started at http://127.0.0.1:35967/ using document root <none> and password file <none>
I20260620 12:23:47.223028 21080 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:47.223084 21080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:47.223291 21080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:47.225463 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "00e584f6726f4b959181ecab1c23d99a"
format_stamp: "Formatted at 2026-06-20 12:23:47 on dist-test-slave-9b3m"
server_key: "71c6b3f94bb6e47132ae447f4ab4ee5e"
server_key_iv: "4014457468768ffaa0957d6a87247229"
server_key_version: "encryptionkey@0"
I20260620 12:23:47.226009 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "00e584f6726f4b959181ecab1c23d99a"
format_stamp: "Formatted at 2026-06-20 12:23:47 on dist-test-slave-9b3m"
server_key: "71c6b3f94bb6e47132ae447f4ab4ee5e"
server_key_iv: "4014457468768ffaa0957d6a87247229"
server_key_version: "encryptionkey@0"
I20260620 12:23:47.229718 21080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:23:47.232091 21095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:47.233479 21080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:47.233605 21080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "00e584f6726f4b959181ecab1c23d99a"
format_stamp: "Formatted at 2026-06-20 12:23:47 on dist-test-slave-9b3m"
server_key: "71c6b3f94bb6e47132ae447f4ab4ee5e"
server_key_iv: "4014457468768ffaa0957d6a87247229"
server_key_version: "encryptionkey@0"
I20260620 12:23:47.233708 21080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:47.244248 21080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:47.247385 21080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:47.247588 21080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:47.254834 21080 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34285
I20260620 12:23:47.254860 21147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34285 every 8 connection(s)
I20260620 12:23:47.255733 21080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:47.258399 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:47.263731 21148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a: Bootstrap starting.
I20260620 12:23:47.264015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21080
I20260620 12:23:47.264166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:47.264431 392 external_mini_cluster.cc:1428] Setting key 5bec99d3619cce5b18846e55609ec474
I20260620 12:23:47.266389 21148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:47.267242 21148 log.cc:824] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:47.269455 21148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a: No bootstrap required, opened a new log
I20260620 12:23:47.273590 21148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e584f6726f4b959181ecab1c23d99a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } }
I20260620 12:23:47.273857 21148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:47.273922 21148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e584f6726f4b959181ecab1c23d99a, State: Initialized, Role: FOLLOWER
I20260620 12:23:47.274402 21148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e584f6726f4b959181ecab1c23d99a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } }
I20260620 12:23:47.274556 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:47.265822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54101 (local address 127.0.0.1:34285)
0620 12:23:47.266223 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:47.266243 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:47.266300 (+ 57us) server_negotiation.cc:407] Connection header received
0620 12:23:47.267163 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:47.267186 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:47.267630 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:47.267922 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:47.268344 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:47.269775 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:47.270730 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:47.271437 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:47.271594 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:47.272458 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:47.272464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:47.272480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:47.272986 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:47.273073 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:47.273105 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:47.273201 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:47.273383 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 12:23:47.273582 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
I20260620 12:23:47.274564 21148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:47.274653 21148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:47.274735 21148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:47.275636 21148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e584f6726f4b959181ecab1c23d99a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } }
I20260620 12:23:47.275983 21148 leader_election.cc:302] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00e584f6726f4b959181ecab1c23d99a; no voters:
I20260620 12:23:47.276319 21148 leader_election.cc:288] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:47.276424 21153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:47.276669 21153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [term 1 LEADER]: Becoming Leader. State: Replica: 00e584f6726f4b959181ecab1c23d99a, State: Running, Role: LEADER
I20260620 12:23:47.277015 21153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e584f6726f4b959181ecab1c23d99a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } }
I20260620 12:23:47.277523 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:47.277669 21148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:47.280669 21154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e584f6726f4b959181ecab1c23d99a. Latest consensus state: current_term: 1 leader_uuid: "00e584f6726f4b959181ecab1c23d99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e584f6726f4b959181ecab1c23d99a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } } }
I20260620 12:23:47.280715 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00e584f6726f4b959181ecab1c23d99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e584f6726f4b959181ecab1c23d99a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } } }
I20260620 12:23:47.280857 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:47.281131 21154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a [sys.catalog]: This master's current role is: LEADER
W20260620 12:23:47.283079 21173 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:23:47.283159 21173 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:23:47.283221 21174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:47.284013 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:47.279806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41573 (local address 127.0.0.1:34285)
0620 12:23:47.279914 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:47.279918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:47.280131 (+ 213us) server_negotiation.cc:407] Connection header received
0620 12:23:47.280560 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:47.280565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:47.280615 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:47.280668 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:47.281113 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:47.281649 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:47.282438 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:47.282668 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:47.282779 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:47.283566 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:47.283570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:47.283572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:47.283584 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:47.283616 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:47.283627 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:47.283703 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:47.283814 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 12:23:47.283860 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:23:47.284104 21174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:47.289420 21174 catalog_manager.cc:1347] Generated new cluster ID: 3cfe8d6756094246a7f143acc5c2e7e7
I20260620 12:23:47.289512 21174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:47.309891 21174 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:47.310838 21174 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:47.320221 21174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00e584f6726f4b959181ecab1c23d99a: Generated new TSK 0
I20260620 12:23:47.321022 21174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:47.330298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21080
2026-06-20T12:23:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:47 dist-test-slave-9b3m krb5kdc[21179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:47 dist-test-slave-9b3m krb5kdc[21179](info): set up 2 sockets
Jun 20 12:23:47 dist-test-slave-9b3m krb5kdc[21179](info): commencing operation
krb5kdc: starting...
W20260620 12:23:48.365166 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:48 dist-test-slave-9b3m krb5kdc[21179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:48.484696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:48.590741 21192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:48.590986 21192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:48.591027 21192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:48.594862 21192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:48.594909 21192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:48.594928 21192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:48.594944 21192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:48.594959 21192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:48.599965 21192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:48.600860 21192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:48.601796 21192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:48.610080 21200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:48.610229 21197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:48.610292 21192 server_base.cc:1034] running on GCE node
W20260620 12:23:48.610097 21198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:48.610924 21192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:48.611984 21192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:48.613150 21192 hybrid_clock.cc:648] HybridClock initialized: now 1781958228613116 us; error 50 us; skew 500 ppm
Jun 20 12:23:48 dist-test-slave-9b3m krb5kdc[21179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:48.615900 21192 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:48.616734 21192 webserver.cc:460] Webserver started at http://127.0.0.1:41931/ using document root <none> and password file <none>
I20260620 12:23:48.617355 21192 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:48.617444 21192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:48.617667 21192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:48.619632 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "0538478a5a684c91a79839e64d558cbe"
format_stamp: "Formatted at 2026-06-20 12:23:48 on dist-test-slave-9b3m"
server_key: "39729abe2f2abf871e8bf32847e2873d"
server_key_iv: "ac70c504390865ec150ff977b87cb3c7"
server_key_version: "encryptionkey@0"
I20260620 12:23:48.620242 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "0538478a5a684c91a79839e64d558cbe"
format_stamp: "Formatted at 2026-06-20 12:23:48 on dist-test-slave-9b3m"
server_key: "39729abe2f2abf871e8bf32847e2873d"
server_key_iv: "ac70c504390865ec150ff977b87cb3c7"
server_key_version: "encryptionkey@0"
I20260620 12:23:48.623908 21192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:23:48.626658 21207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:48.628033 21192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 12:23:48.628180 21192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "0538478a5a684c91a79839e64d558cbe"
format_stamp: "Formatted at 2026-06-20 12:23:48 on dist-test-slave-9b3m"
server_key: "39729abe2f2abf871e8bf32847e2873d"
server_key_iv: "ac70c504390865ec150ff977b87cb3c7"
server_key_version: "encryptionkey@0"
I20260620 12:23:48.628294 21192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:48.632891 21192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:48.635864 21192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:48.636132 21192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:48.643437 21192 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33997
I20260620 12:23:48.643445 21259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33997 every 8 connection(s)
I20260620 12:23:48.644469 21192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:48.647274 21260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:48.650413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21192
I20260620 12:23:48.650593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:48.650882 392 external_mini_cluster.cc:1428] Setting key 1358b094050095ad34a1d9026dc8ad17
I20260620 12:23:48.653512 21260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe: Bootstrap starting.
I20260620 12:23:48.656131 21260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:48.656224 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:48.652138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43617 (local address 127.0.0.1:33997)
0620 12:23:48.652557 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:48.652577 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:48.652619 (+ 42us) server_negotiation.cc:407] Connection header received
0620 12:23:48.653433 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:48.653454 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:48.653805 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:48.654155 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:48.654891 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:48.654896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:48.654910 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:48.654922 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:48.654982 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:48.655008 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:48.655149 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:48.655212 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 12:23:48.655351 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260620 12:23:48.658320 21260 log.cc:824] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:48.659248 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:48.660331 21260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe: No bootstrap required, opened a new log
I20260620 12:23:48.663380 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:48.661607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50155 (local address 127.0.0.1:33997)
0620 12:23:48.661779 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:48.661784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:48.661919 (+ 135us) server_negotiation.cc:407] Connection header received
0620 12:23:48.662033 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:48.662036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:48.662069 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:48.662169 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:48.662976 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:48.662980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:48.662983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:48.662996 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:48.663025 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:48.663039 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:48.663137 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:48.663226 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 12:23:48.663263 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 12:23:48.665490 21260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0538478a5a684c91a79839e64d558cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } }
I20260620 12:23:48.665752 21260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:48.665858 21260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0538478a5a684c91a79839e64d558cbe, State: Initialized, Role: FOLLOWER
I20260620 12:23:48.666369 21260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0538478a5a684c91a79839e64d558cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } }
I20260620 12:23:48.666579 21260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:48.666684 21260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:48.666823 21260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:48.668167 21260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0538478a5a684c91a79839e64d558cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } }
I20260620 12:23:48.668524 21260 leader_election.cc:302] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0538478a5a684c91a79839e64d558cbe; no voters:
I20260620 12:23:48.668860 21260 leader_election.cc:288] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:48.668936 21270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:48.669193 21270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [term 1 LEADER]: Becoming Leader. State: Replica: 0538478a5a684c91a79839e64d558cbe, State: Running, Role: LEADER
I20260620 12:23:48.669517 21270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0538478a5a684c91a79839e64d558cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } }
I20260620 12:23:48.670231 21260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:48.670576 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0538478a5a684c91a79839e64d558cbe. Latest consensus state: current_term: 1 leader_uuid: "0538478a5a684c91a79839e64d558cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0538478a5a684c91a79839e64d558cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } } }
I20260620 12:23:48.670686 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:48.670909 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0538478a5a684c91a79839e64d558cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0538478a5a684c91a79839e64d558cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } } }
I20260620 12:23:48.670992 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:48.671330 21274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:48.674356 21274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:48.679816 21274 catalog_manager.cc:1347] Generated new cluster ID: 85fe81e883704856af8467e9b597238a
I20260620 12:23:48.679889 21274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:48.693774 21274 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:48.694161 21274 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:48.697832 21274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0538478a5a684c91a79839e64d558cbe: Generated new TSK 0
I20260620 12:23:48.698436 21274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:48.707296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21192
2026-06-20T12:23:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:48 dist-test-slave-9b3m krb5kdc[21291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:48 dist-test-slave-9b3m krb5kdc[21291](info): set up 2 sockets
Jun 20 12:23:48 dist-test-slave-9b3m krb5kdc[21291](info): commencing operation
krb5kdc: starting...
W20260620 12:23:49.765350 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:49 dist-test-slave-9b3m krb5kdc[21291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:49.883621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:49.990821 21304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:49.991097 21304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:49.991168 21304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:49.995126 21304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:49.995213 21304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:49.995250 21304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:49.995308 21304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:49.995335 21304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:50.000602 21304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:50.001770 21304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:50.002805 21304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:50.011340 21309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:50.011545 21304 server_base.cc:1034] running on GCE node
W20260620 12:23:50.011360 21312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:50.011353 21310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:50.012200 21304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:50.013370 21304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:50.014551 21304 hybrid_clock.cc:648] HybridClock initialized: now 1781958230014519 us; error 46 us; skew 500 ppm
Jun 20 12:23:50 dist-test-slave-9b3m krb5kdc[21291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:50.017275 21304 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:50.018172 21304 webserver.cc:460] Webserver started at http://127.0.0.1:34795/ using document root <none> and password file <none>
I20260620 12:23:50.018740 21304 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:50.018832 21304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:50.019063 21304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:50.021157 21304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "f09d303974294e5185bb5896ae894571"
format_stamp: "Formatted at 2026-06-20 12:23:50 on dist-test-slave-9b3m"
server_key: "f54e857686c7ae6b401b0c9428864c57"
server_key_iv: "7c2aed3816ab63cd02d6122c58f4af90"
server_key_version: "encryptionkey@0"
I20260620 12:23:50.021735 21304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "f09d303974294e5185bb5896ae894571"
format_stamp: "Formatted at 2026-06-20 12:23:50 on dist-test-slave-9b3m"
server_key: "f54e857686c7ae6b401b0c9428864c57"
server_key_iv: "7c2aed3816ab63cd02d6122c58f4af90"
server_key_version: "encryptionkey@0"
I20260620 12:23:50.025377 21304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 12:23:50.027706 21319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:50.028920 21304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:50.029102 21304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "f09d303974294e5185bb5896ae894571"
format_stamp: "Formatted at 2026-06-20 12:23:50 on dist-test-slave-9b3m"
server_key: "f54e857686c7ae6b401b0c9428864c57"
server_key_iv: "7c2aed3816ab63cd02d6122c58f4af90"
server_key_version: "encryptionkey@0"
I20260620 12:23:50.029214 21304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:50.053007 21304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:50.055845 21304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:50.056051 21304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:50.063005 21304 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45895
I20260620 12:23:50.063032 21371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45895 every 8 connection(s)
I20260620 12:23:50.063905 21304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:50.066576 21372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:50.069619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21304
I20260620 12:23:50.069761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:50.070072 392 external_mini_cluster.cc:1428] Setting key df64af5caced84416a3126be02ac667d
I20260620 12:23:50.072283 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571: Bootstrap starting.
I20260620 12:23:50.074441 21372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:50.075162 21372 log.cc:824] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:50.076916 21372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571: No bootstrap required, opened a new log
Jun 20 12:23:50 dist-test-slave-9b3m krb5kdc[21291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:23:50.080994 21372 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09d303974294e5185bb5896ae894571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260620 12:23:50.081257 21372 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:50.081346 21372 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09d303974294e5185bb5896ae894571, State: Initialized, Role: FOLLOWER
I20260620 12:23:50.081825 21372 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09d303974294e5185bb5896ae894571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260620 12:23:50.081967 21372 raft_consensus.cc:388] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:50.082029 21372 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:50.082147 21372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:50.082978 21372 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09d303974294e5185bb5896ae894571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260620 12:23:50.083302 21372 leader_election.cc:302] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09d303974294e5185bb5896ae894571; no voters:
I20260620 12:23:50.083642 21372 leader_election.cc:288] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:50.083724 21377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:50.083922 21377 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [term 1 LEADER]: Becoming Leader. State: Replica: f09d303974294e5185bb5896ae894571, State: Running, Role: LEADER
I20260620 12:23:50.084626 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:50.071423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:40511 (local address 127.0.0.1:45895)
0620 12:23:50.071825 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:50.071839 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:50.071873 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:23:50.072538 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:50.072557 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:50.072884 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:50.073274 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:50.074078 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:50.075118 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:50.076184 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:50.076640 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:50.076817 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:50.079507 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:50.079512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:50.079529 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:50.079555 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:50.081379 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:50.081900 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:50.081906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:50.081913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:50.082073 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:50.082405 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:50.082408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:50.082410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:50.082771 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:50.083004 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:50.083319 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 12:23:50.083537 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260620 12:23:50.084839 21372 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:50.084884 21377 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09d303974294e5185bb5896ae894571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260620 12:23:50.086462 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f09d303974294e5185bb5896ae894571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09d303974294e5185bb5896ae894571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } } }
I20260620 12:23:50.086596 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:50.086956 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09d303974294e5185bb5896ae894571. Latest consensus state: current_term: 1 leader_uuid: "f09d303974294e5185bb5896ae894571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09d303974294e5185bb5896ae894571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } } }
I20260620 12:23:50.087821 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:50.087850 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 12:23:50.090205 21392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:23:50.090268 21392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:23:50.090351 21391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:50.091320 21391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:50.097136 21391 catalog_manager.cc:1347] Generated new cluster ID: 02939adb41b749a3aabbb5e2f9ffc2d3
I20260620 12:23:50.097231 21391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:50.098757 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:50.089788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:52175 (local address 127.0.0.1:45895)
0620 12:23:50.089883 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:50.089887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:50.090650 (+ 763us) server_negotiation.cc:407] Connection header received
0620 12:23:50.090767 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:50.090770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:50.090816 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:50.090894 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:50.091715 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:50.092494 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:50.093453 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:50.093675 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:50.093780 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:50.095224 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:50.095228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:50.095230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:50.095257 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:50.096737 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:50.097420 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:50.097424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:50.097426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:50.097511 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:50.097892 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:50.097895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:50.097897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:50.098101 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:50.098244 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:50.098510 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 12:23:50.098608 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 12:23:50.117347 21391 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:50.118251 21391 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:50.126159 21391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f09d303974294e5185bb5896ae894571: Generated new TSK 0
I20260620 12:23:50.126803 21391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:50.132356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21304
W20260620 12:23:50.137068 21289 connection.cc:535] client connection to 127.0.0.1:45895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45895): Connection reset by peer (error 104)
2026-06-20T12:23:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:50 dist-test-slave-9b3m krb5kdc[21403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:50 dist-test-slave-9b3m krb5kdc[21403](info): set up 2 sockets
Jun 20 12:23:50 dist-test-slave-9b3m krb5kdc[21403](info): commencing operation
krb5kdc: starting...
W20260620 12:23:51.161342 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:51 dist-test-slave-9b3m krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:51.280516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:51.388548 21416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:51.388839 21416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:51.388914 21416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:51.392844 21416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:51.392938 21416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:51.392978 21416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:51.393038 21416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:51.393090 21416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:51.398285 21416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:51.399335 21416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:51.400393 21416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:51.408643 21424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:51.408650 21421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:51.408689 21422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:51.409022 21416 server_base.cc:1034] running on GCE node
I20260620 12:23:51.409570 21416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:51.410696 21416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:51.411902 21416 hybrid_clock.cc:648] HybridClock initialized: now 1781958231411882 us; error 31 us; skew 500 ppm
Jun 20 12:23:51 dist-test-slave-9b3m krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:51.419484 21416 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:51.420434 21416 webserver.cc:460] Webserver started at http://127.0.0.1:45405/ using document root <none> and password file <none>
I20260620 12:23:51.421011 21416 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:51.421130 21416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:51.421386 21416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:51.423424 21416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "5ec182406fcb41e1b3724b4fe066c715"
format_stamp: "Formatted at 2026-06-20 12:23:51 on dist-test-slave-9b3m"
server_key: "a3cf433ca31c219f6ba4af539f0e017b"
server_key_iv: "68d06360dadd66c3640606ae95354176"
server_key_version: "encryptionkey@0"
I20260620 12:23:51.423981 21416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ec182406fcb41e1b3724b4fe066c715"
format_stamp: "Formatted at 2026-06-20 12:23:51 on dist-test-slave-9b3m"
server_key: "a3cf433ca31c219f6ba4af539f0e017b"
server_key_iv: "68d06360dadd66c3640606ae95354176"
server_key_version: "encryptionkey@0"
I20260620 12:23:51.427774 21416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:23:51.430392 21431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:51.431669 21416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:23:51.431820 21416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "5ec182406fcb41e1b3724b4fe066c715"
format_stamp: "Formatted at 2026-06-20 12:23:51 on dist-test-slave-9b3m"
server_key: "a3cf433ca31c219f6ba4af539f0e017b"
server_key_iv: "68d06360dadd66c3640606ae95354176"
server_key_version: "encryptionkey@0"
I20260620 12:23:51.431950 21416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:51.446338 21416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:51.449570 21416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:51.449795 21416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:51.456770 21416 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32889
I20260620 12:23:51.456859 21483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32889 every 8 connection(s)
I20260620 12:23:51.457763 21416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:51.460475 21484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:51.465737 21484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715: Bootstrap starting.
I20260620 12:23:51.466876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21416
I20260620 12:23:51.467031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:51.467386 392 external_mini_cluster.cc:1428] Setting key 89e5691689360bb5418e8579b5242b51
I20260620 12:23:51.468575 21484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:51.469398 21484 log.cc:824] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:51.471290 21484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715: No bootstrap required, opened a new log
I20260620 12:23:51.474885 21484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec182406fcb41e1b3724b4fe066c715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260620 12:23:51.475090 21484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:51.475227 21484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec182406fcb41e1b3724b4fe066c715, State: Initialized, Role: FOLLOWER
I20260620 12:23:51.475697 21484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec182406fcb41e1b3724b4fe066c715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260620 12:23:51.475821 21484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:51.475888 21484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:51.476016 21484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:51.476501 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:51.468485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:59647 (local address 127.0.0.1:32889)
0620 12:23:51.468923 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:51.468938 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:51.468973 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:23:51.469770 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:51.469795 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:51.470226 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:51.470562 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:51.470958 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:51.471977 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:51.472877 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:51.473432 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:51.473606 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:51.474542 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:51.474547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:51.474560 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:51.474949 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:51.475001 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:51.475023 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:51.475179 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:51.475444 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 12:23:51.475586 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":144,"threads_started":1}
I20260620 12:23:51.476944 21484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec182406fcb41e1b3724b4fe066c715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260620 12:23:51.477296 21484 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec182406fcb41e1b3724b4fe066c715; no voters:
I20260620 12:23:51.477644 21484 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:51.477725 21489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:51.477942 21489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec182406fcb41e1b3724b4fe066c715, State: Running, Role: LEADER
I20260620 12:23:51.478273 21489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec182406fcb41e1b3724b4fe066c715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260620 12:23:51.478896 21484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:51.479738 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:51.480026 21491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ec182406fcb41e1b3724b4fe066c715. Latest consensus state: current_term: 1 leader_uuid: "5ec182406fcb41e1b3724b4fe066c715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec182406fcb41e1b3724b4fe066c715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } } }
I20260620 12:23:51.480139 21491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:51.480430 21490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ec182406fcb41e1b3724b4fe066c715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec182406fcb41e1b3724b4fe066c715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } } }
I20260620 12:23:51.480504 21490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:51.480526 21498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:51.483587 21498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:51.487212 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:51.481997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:55363 (local address 127.0.0.1:32889)
0620 12:23:51.482101 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:51.482105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:51.482715 (+ 610us) server_negotiation.cc:407] Connection header received
0620 12:23:51.482838 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:51.482843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:51.482887 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:51.482982 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:51.483245 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:51.484021 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:51.484946 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:51.485313 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:51.485396 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:51.486665 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:51.486669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:51.486671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:51.486685 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:51.486714 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:51.486727 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:51.486785 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:51.487034 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 12:23:51.487070 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:23:51.489398 21498 catalog_manager.cc:1347] Generated new cluster ID: 57cbbc3756bb4c6cbc64a89b7c4079fc
I20260620 12:23:51.489460 21498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:51.504365 21498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:51.505316 21498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:51.522495 21498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ec182406fcb41e1b3724b4fe066c715: Generated new TSK 0
I20260620 12:23:51.523365 21498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:51.535977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21416
W20260620 12:23:51.541566 21401 connection.cc:535] client connection to 127.0.0.1:32889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32889): Connection reset by peer (error 104)
2026-06-20T12:23:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:51 dist-test-slave-9b3m krb5kdc[21515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:51 dist-test-slave-9b3m krb5kdc[21515](info): set up 2 sockets
Jun 20 12:23:51 dist-test-slave-9b3m krb5kdc[21515](info): commencing operation
krb5kdc: starting...
W20260620 12:23:52.581411 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:52 dist-test-slave-9b3m krb5kdc[21515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:52.700156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:52.805780 21528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:52.806008 21528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:52.806047 21528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:52.809847 21528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:52.809894 21528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:52.809913 21528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:52.809929 21528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:52.809945 21528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:52.814985 21528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:52.815883 21528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:52.816779 21528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:52.824985 21534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:52.825021 21533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:52.825026 21536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:52.825205 21528 server_base.cc:1034] running on GCE node
I20260620 12:23:52.825831 21528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:52.826855 21528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:52.828042 21528 hybrid_clock.cc:648] HybridClock initialized: now 1781958232828015 us; error 44 us; skew 500 ppm
Jun 20 12:23:52 dist-test-slave-9b3m krb5kdc[21515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:52.830806 21528 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:52.831636 21528 webserver.cc:460] Webserver started at http://127.0.0.1:33737/ using document root <none> and password file <none>
I20260620 12:23:52.832185 21528 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:52.832276 21528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:52.832504 21528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:52.834582 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "1836d357cfb643c2bf825ee5fccc7fc5"
format_stamp: "Formatted at 2026-06-20 12:23:52 on dist-test-slave-9b3m"
server_key: "33b49bb8714ae60bbbe314df2617a79b"
server_key_iv: "c9a908fe6c85965c2c7bc1b9934a8dd5"
server_key_version: "encryptionkey@0"
I20260620 12:23:52.835162 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "1836d357cfb643c2bf825ee5fccc7fc5"
format_stamp: "Formatted at 2026-06-20 12:23:52 on dist-test-slave-9b3m"
server_key: "33b49bb8714ae60bbbe314df2617a79b"
server_key_iv: "c9a908fe6c85965c2c7bc1b9934a8dd5"
server_key_version: "encryptionkey@0"
I20260620 12:23:52.838882 21528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:23:52.841181 21543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:52.842355 21528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:52.842494 21528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "1836d357cfb643c2bf825ee5fccc7fc5"
format_stamp: "Formatted at 2026-06-20 12:23:52 on dist-test-slave-9b3m"
server_key: "33b49bb8714ae60bbbe314df2617a79b"
server_key_iv: "c9a908fe6c85965c2c7bc1b9934a8dd5"
server_key_version: "encryptionkey@0"
I20260620 12:23:52.842620 21528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:52.847162 21528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:52.850036 21528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:52.850229 21528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:52.856873 21528 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39461
I20260620 12:23:52.856882 21595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39461 every 8 connection(s)
I20260620 12:23:52.857792 21528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:52.860374 21596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:52.865593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21528
I20260620 12:23:52.865731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:52.865756 21596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5: Bootstrap starting.
I20260620 12:23:52.866046 392 external_mini_cluster.cc:1428] Setting key 199eb1925b60cc2191c93ef50c3d8db1
I20260620 12:23:52.868259 21596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:52.869062 21596 log.cc:824] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:52.870924 21596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5: No bootstrap required, opened a new log
I20260620 12:23:52.871073 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:52.867246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:47883 (local address 127.0.0.1:39461)
0620 12:23:52.867659 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:52.867672 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:52.867707 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:23:52.868333 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:52.868356 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:52.868654 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:52.868924 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:52.869847 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:52.869851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:52.869862 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:52.869872 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:52.869923 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:52.869942 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:52.870075 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:52.870163 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 12:23:52.870321 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":130,"threads_started":1}
I20260620 12:23:52.873975 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:52.875082 21596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1836d357cfb643c2bf825ee5fccc7fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39461 } }
I20260620 12:23:52.875373 21596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:52.875459 21596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1836d357cfb643c2bf825ee5fccc7fc5, State: Initialized, Role: FOLLOWER
I20260620 12:23:52.875897 21596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1836d357cfb643c2bf825ee5fccc7fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39461 } }
I20260620 12:23:52.876127 21596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:52.876245 21596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:52.876363 21596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:52.877625 21596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1836d357cfb643c2bf825ee5fccc7fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39461 } }
I20260620 12:23:52.877877 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:52.876135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:49273 (local address 127.0.0.1:39461)
0620 12:23:52.876260 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:52.876264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:52.876522 (+ 258us) server_negotiation.cc:407] Connection header received
0620 12:23:52.876663 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:52.876665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:52.876695 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:52.876758 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:52.877464 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:52.877467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:52.877469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:52.877479 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:52.877502 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:52.877511 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:52.877579 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:52.877696 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 12:23:52.877753 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:23:52.878010 21596 leader_election.cc:302] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1836d357cfb643c2bf825ee5fccc7fc5; no voters:
I20260620 12:23:52.878367 21596 leader_election.cc:288] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:52.878448 21606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:52.878655 21606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [term 1 LEADER]: Becoming Leader. State: Replica: 1836d357cfb643c2bf825ee5fccc7fc5, State: Running, Role: LEADER
I20260620 12:23:52.879005 21606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1836d357cfb643c2bf825ee5fccc7fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39461 } }
I20260620 12:23:52.879570 21596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:52.880167 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1836d357cfb643c2bf825ee5fccc7fc5. Latest consensus state: current_term: 1 leader_uuid: "1836d357cfb643c2bf825ee5fccc7fc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1836d357cfb643c2bf825ee5fccc7fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39461 } } }
I20260620 12:23:52.880362 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:52.880672 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1836d357cfb643c2bf825ee5fccc7fc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1836d357cfb643c2bf825ee5fccc7fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39461 } } }
I20260620 12:23:52.880775 21615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:52.880844 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:52.883989 21615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:52.889127 21615 catalog_manager.cc:1347] Generated new cluster ID: 045262f42f9740aebc49adf5610410e6
I20260620 12:23:52.889191 21615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:52.896546 21615 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:52.896842 21615 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:52.904244 21615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1836d357cfb643c2bf825ee5fccc7fc5: Generated new TSK 0
I20260620 12:23:52.904920 21615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:52.910806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21528
2026-06-20T12:23:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:52 dist-test-slave-9b3m krb5kdc[21627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:52 dist-test-slave-9b3m krb5kdc[21627](info): set up 2 sockets
Jun 20 12:23:52 dist-test-slave-9b3m krb5kdc[21627](info): commencing operation
krb5kdc: starting...
W20260620 12:23:53.965221 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:54 dist-test-slave-9b3m krb5kdc[21627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:54.088624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:54.196724 21640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:54.196964 21640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:54.197005 21640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:54.201021 21640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:54.201130 21640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:54.201181 21640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:54.201213 21640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:54.201259 21640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:54.206507 21640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:54.207469 21640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:54.208433 21640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:54.216660 21646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:54.216665 21648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:54.216743 21645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:54.216862 21640 server_base.cc:1034] running on GCE node
I20260620 12:23:54.217456 21640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:54.218493 21640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:54.219651 21640 hybrid_clock.cc:648] HybridClock initialized: now 1781958234219624 us; error 41 us; skew 500 ppm
Jun 20 12:23:54 dist-test-slave-9b3m krb5kdc[21627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:54.222517 21640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:54.223407 21640 webserver.cc:460] Webserver started at http://127.0.0.1:35775/ using document root <none> and password file <none>
I20260620 12:23:54.223945 21640 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:54.224005 21640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:54.224155 21640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:54.226099 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "7a07f9a460444835bb1d4b29d23a1eb4"
format_stamp: "Formatted at 2026-06-20 12:23:54 on dist-test-slave-9b3m"
server_key: "60052af108eb7cc57e8f74cebf8c8962"
server_key_iv: "5e9c6c8f3274b3b1ebbd9cc95f283a48"
server_key_version: "encryptionkey@0"
I20260620 12:23:54.226599 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a07f9a460444835bb1d4b29d23a1eb4"
format_stamp: "Formatted at 2026-06-20 12:23:54 on dist-test-slave-9b3m"
server_key: "60052af108eb7cc57e8f74cebf8c8962"
server_key_iv: "5e9c6c8f3274b3b1ebbd9cc95f283a48"
server_key_version: "encryptionkey@0"
I20260620 12:23:54.230095 21640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:23:54.232354 21655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:54.233664 21640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:54.233788 21640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "7a07f9a460444835bb1d4b29d23a1eb4"
format_stamp: "Formatted at 2026-06-20 12:23:54 on dist-test-slave-9b3m"
server_key: "60052af108eb7cc57e8f74cebf8c8962"
server_key_iv: "5e9c6c8f3274b3b1ebbd9cc95f283a48"
server_key_version: "encryptionkey@0"
I20260620 12:23:54.233877 21640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:54.268333 21640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:54.271500 21640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:54.271649 21640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:54.278638 21640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38827
I20260620 12:23:54.278666 21707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38827 every 8 connection(s)
I20260620 12:23:54.279556 21640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:54.282162 21708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:54.284550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21640
I20260620 12:23:54.284695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:54.284988 392 external_mini_cluster.cc:1428] Setting key 4a2f00db22c156ef54a55ee495a6a348
I20260620 12:23:54.288252 21708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4: Bootstrap starting.
I20260620 12:23:54.290800 21708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:54.291527 21708 log.cc:824] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:54.293457 21708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4: No bootstrap required, opened a new log
Jun 20 12:23:54 dist-test-slave-9b3m krb5kdc[21627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 12:23:54.297379 21708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a07f9a460444835bb1d4b29d23a1eb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } }
I20260620 12:23:54.297647 21708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:54.297729 21708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a07f9a460444835bb1d4b29d23a1eb4, State: Initialized, Role: FOLLOWER
I20260620 12:23:54.298183 21708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a07f9a460444835bb1d4b29d23a1eb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } }
I20260620 12:23:54.298377 21708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:54.298458 21708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:54.298568 21708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:54.299412 21708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a07f9a460444835bb1d4b29d23a1eb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } }
I20260620 12:23:54.299770 21708 leader_election.cc:302] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a07f9a460444835bb1d4b29d23a1eb4; no voters:
I20260620 12:23:54.300077 21708 leader_election.cc:288] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:54.300179 21713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:54.300428 21713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [term 1 LEADER]: Becoming Leader. State: Replica: 7a07f9a460444835bb1d4b29d23a1eb4, State: Running, Role: LEADER
I20260620 12:23:54.300769 21713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a07f9a460444835bb1d4b29d23a1eb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } }
I20260620 12:23:54.300908 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:54.286380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:55657 (local address 127.0.0.1:38827)
0620 12:23:54.286780 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:54.286799 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:54.286840 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:23:54.287615 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:54.287636 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:54.288041 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:54.288404 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:54.289102 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:54.290555 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:54.291533 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:54.292120 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:54.292321 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:54.294820 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:54.294827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:54.294849 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:54.294878 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:54.297514 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:54.298148 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:54.298153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:54.298159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:54.298370 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:54.298758 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:54.298768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:54.298770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:54.299162 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:54.299405 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:54.299701 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 12:23:54.299918 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260620 12:23:54.301311 21708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:54.302420 21714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a07f9a460444835bb1d4b29d23a1eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a07f9a460444835bb1d4b29d23a1eb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } } }
I20260620 12:23:54.302548 21714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:54.302886 21715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a07f9a460444835bb1d4b29d23a1eb4. Latest consensus state: current_term: 1 leader_uuid: "7a07f9a460444835bb1d4b29d23a1eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a07f9a460444835bb1d4b29d23a1eb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } } }
I20260620 12:23:54.302968 21715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:54.303269 21722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:54.304196 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:54.307174 21722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:54.313645 21722 catalog_manager.cc:1347] Generated new cluster ID: f7ac7d38a4cf4192b934718482c213db
I20260620 12:23:54.313712 21722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:54.315935 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:54.306590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:57619 (local address 127.0.0.1:38827)
0620 12:23:54.306680 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:54.306734 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:54.306749 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:23:54.306785 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:54.306789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:54.306830 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:54.306912 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:54.307713 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:54.308492 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:54.309617 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:54.309983 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:54.310099 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:54.312161 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:54.312165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:54.312168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:54.312193 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:54.314089 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:54.314638 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:54.314642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:54.314643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:54.314739 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:54.315069 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:54.315073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:54.315075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:54.315276 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:54.315431 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:54.315682 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 12:23:54.315783 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 12:23:54.349988 21722 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:54.351006 21722 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:54.359673 21722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a07f9a460444835bb1d4b29d23a1eb4: Generated new TSK 0
I20260620 12:23:54.360378 21722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:54.364068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21640
W20260620 12:23:54.369553 21625 connection.cc:535] client connection to 127.0.0.1:38827 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38827): Connection reset by peer (error 104)
2026-06-20T12:23:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:54 dist-test-slave-9b3m krb5kdc[21739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:54 dist-test-slave-9b3m krb5kdc[21739](info): set up 2 sockets
Jun 20 12:23:54 dist-test-slave-9b3m krb5kdc[21739](info): commencing operation
krb5kdc: starting...
W20260620 12:23:55.409355 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:55 dist-test-slave-9b3m krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:55.532330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:55.640776 21752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:55.641021 21752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:55.641083 21752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:55.645006 21752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:55.645136 21752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:55.645164 21752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:55.645200 21752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:55.645233 21752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:55.650578 21752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:55.651600 21752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:55.652602 21752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:55.661108 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:55.661094 21757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:55.661094 21760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:55.661331 21752 server_base.cc:1034] running on GCE node
I20260620 12:23:55.662045 21752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:55.663096 21752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:55.664276 21752 hybrid_clock.cc:648] HybridClock initialized: now 1781958235664255 us; error 33 us; skew 500 ppm
Jun 20 12:23:55 dist-test-slave-9b3m krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:55.667078 21752 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:55.667925 21752 webserver.cc:460] Webserver started at http://127.0.0.1:38107/ using document root <none> and password file <none>
I20260620 12:23:55.668412 21752 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:55.668466 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:55.668622 21752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:55.670591 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "3fb752c199994c7a91756ab2621c580e"
format_stamp: "Formatted at 2026-06-20 12:23:55 on dist-test-slave-9b3m"
server_key: "02aa160a55967986c3b3c4462429bdea"
server_key_iv: "934c852b8e84cec95804c2f23a17bc83"
server_key_version: "encryptionkey@0"
I20260620 12:23:55.671079 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "3fb752c199994c7a91756ab2621c580e"
format_stamp: "Formatted at 2026-06-20 12:23:55 on dist-test-slave-9b3m"
server_key: "02aa160a55967986c3b3c4462429bdea"
server_key_iv: "934c852b8e84cec95804c2f23a17bc83"
server_key_version: "encryptionkey@0"
I20260620 12:23:55.674510 21752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 12:23:55.676779 21767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:55.678089 21752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:55.678217 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "3fb752c199994c7a91756ab2621c580e"
format_stamp: "Formatted at 2026-06-20 12:23:55 on dist-test-slave-9b3m"
server_key: "02aa160a55967986c3b3c4462429bdea"
server_key_iv: "934c852b8e84cec95804c2f23a17bc83"
server_key_version: "encryptionkey@0"
I20260620 12:23:55.678319 21752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:55.701817 21752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:55.705104 21752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:55.705308 21752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:55.712788 21752 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41063
I20260620 12:23:55.712860 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41063 every 8 connection(s)
I20260620 12:23:55.713793 21752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:55.716471 21820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:55.718377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21752
I20260620 12:23:55.718520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:55.718806 392 external_mini_cluster.cc:1428] Setting key 28803c207fbc53ace999ee6c0e0397c0
I20260620 12:23:55.723326 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e: Bootstrap starting.
I20260620 12:23:55.725800 21820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:55.726570 21820 log.cc:824] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:55.728415 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e: No bootstrap required, opened a new log
I20260620 12:23:55.728434 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:55.720238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:42613 (local address 127.0.0.1:41063)
0620 12:23:55.720643 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:55.720662 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:55.720705 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:23:55.721507 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:55.721530 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:55.721969 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:55.722359 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:55.722766 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:55.724176 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:55.725126 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:55.725801 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:55.725965 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:55.726811 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:55.726816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:55.726828 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:55.727198 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:55.727253 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:55.727274 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:55.727445 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:55.727639 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 12:23:55.727771 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260620 12:23:55.731357 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:55.732475 21820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb752c199994c7a91756ab2621c580e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } }
I20260620 12:23:55.732704 21820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:55.732779 21820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb752c199994c7a91756ab2621c580e, State: Initialized, Role: FOLLOWER
I20260620 12:23:55.733281 21820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb752c199994c7a91756ab2621c580e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } }
I20260620 12:23:55.733506 21820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:55.733588 21820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:55.733701 21820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:55.734809 21820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb752c199994c7a91756ab2621c580e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } }
I20260620 12:23:55.735165 21820 leader_election.cc:302] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fb752c199994c7a91756ab2621c580e; no voters:
I20260620 12:23:55.735489 21820 leader_election.cc:288] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:55.735586 21830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:55.735792 21830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [term 1 LEADER]: Becoming Leader. State: Replica: 3fb752c199994c7a91756ab2621c580e, State: Running, Role: LEADER
I20260620 12:23:55.736155 21830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb752c199994c7a91756ab2621c580e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } }
I20260620 12:23:55.736752 21820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:55.737841 21832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fb752c199994c7a91756ab2621c580e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb752c199994c7a91756ab2621c580e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } } }
I20260620 12:23:55.737970 21832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:55.738219 21832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fb752c199994c7a91756ab2621c580e. Latest consensus state: current_term: 1 leader_uuid: "3fb752c199994c7a91756ab2621c580e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb752c199994c7a91756ab2621c580e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41063 } } }
I20260620 12:23:55.738293 21832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:55.738523 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:55.733339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:33781 (local address 127.0.0.1:41063)
0620 12:23:55.733459 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:55.733463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:55.733785 (+ 322us) server_negotiation.cc:407] Connection header received
0620 12:23:55.733826 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:55.733828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:55.733873 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:55.733959 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:55.734228 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:55.734986 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:55.736039 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:55.736359 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:55.736477 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:55.737363 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:55.737367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:55.737369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:55.737383 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:55.737410 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:55.737424 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:55.737488 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:55.737768 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 12:23:55.737809 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 12:23:55.741279 21845 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:23:55.741335 21845 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:23:55.741415 21839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:55.742094 21839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:55.747026 21839 catalog_manager.cc:1347] Generated new cluster ID: 57e467030b544945bd5990585ce5366b
I20260620 12:23:55.747104 21839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:55.766736 21839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:55.767680 21839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:55.777318 21839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fb752c199994c7a91756ab2621c580e: Generated new TSK 0
I20260620 12:23:55.778030 21839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:55.786418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21752
W20260620 12:23:55.791988 21737 connection.cc:535] client connection to 127.0.0.1:41063 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41063): Connection reset by peer (error 104)
2026-06-20T12:23:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:55 dist-test-slave-9b3m krb5kdc[21851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:55 dist-test-slave-9b3m krb5kdc[21851](info): set up 2 sockets
Jun 20 12:23:55 dist-test-slave-9b3m krb5kdc[21851](info): commencing operation
krb5kdc: starting...
W20260620 12:23:56.841387 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:56 dist-test-slave-9b3m krb5kdc[21851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 12:23:56.964203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:57.071856 21864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:57.072098 21864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:57.072137 21864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:57.076145 21864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:57.076200 21864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:57.076220 21864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:57.076236 21864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:57.076251 21864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:57.081463 21864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42747
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:57.082420 21864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:57.083379 21864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:57.091537 21870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:57.091545 21872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:57.091612 21864 server_base.cc:1034] running on GCE node
W20260620 12:23:57.091562 21869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:57.092312 21864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:57.093474 21864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:57.094640 21864 hybrid_clock.cc:648] HybridClock initialized: now 1781958237094602 us; error 53 us; skew 500 ppm
Jun 20 12:23:57 dist-test-slave-9b3m krb5kdc[21851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:57.101204 21864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 12:23:57.102145 21864 webserver.cc:460] Webserver started at http://127.0.0.1:34083/ using document root <none> and password file <none>
I20260620 12:23:57.102736 21864 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:57.102829 21864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:57.103061 21864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:57.105088 21864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "1c8b45ad85f94bb89678a3ead878ab81"
format_stamp: "Formatted at 2026-06-20 12:23:57 on dist-test-slave-9b3m"
server_key: "af082416dbf5f043f14e569a4f28f757"
server_key_iv: "6a5ec42e26ea913f212961d2025450db"
server_key_version: "encryptionkey@0"
I20260620 12:23:57.105664 21864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c8b45ad85f94bb89678a3ead878ab81"
format_stamp: "Formatted at 2026-06-20 12:23:57 on dist-test-slave-9b3m"
server_key: "af082416dbf5f043f14e569a4f28f757"
server_key_iv: "6a5ec42e26ea913f212961d2025450db"
server_key_version: "encryptionkey@0"
I20260620 12:23:57.109222 21864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:23:57.111675 21879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:57.113147 21864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:23:57.113328 21864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "1c8b45ad85f94bb89678a3ead878ab81"
format_stamp: "Formatted at 2026-06-20 12:23:57 on dist-test-slave-9b3m"
server_key: "af082416dbf5f043f14e569a4f28f757"
server_key_iv: "6a5ec42e26ea913f212961d2025450db"
server_key_version: "encryptionkey@0"
I20260620 12:23:57.113454 21864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:57.118434 21864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:57.121680 21864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:57.121886 21864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:57.129350 21864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42747
I20260620 12:23:57.129487 21931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42747 every 8 connection(s)
I20260620 12:23:57.130290 21864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:57.133211 21932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:57.138918 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81: Bootstrap starting.
I20260620 12:23:57.139789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21864
I20260620 12:23:57.139886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:57.140195 392 external_mini_cluster.cc:1428] Setting key 85220e3cf1dfda69db647cb06502dd7d
I20260620 12:23:57.141937 21932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:57.142859 21932 log.cc:824] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:57.144759 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81: No bootstrap required, opened a new log
I20260620 12:23:57.145898 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:57.141393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:55749 (local address 127.0.0.1:42747)
0620 12:23:57.141896 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:57.141918 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:57.141965 (+ 47us) server_negotiation.cc:407] Connection header received
0620 12:23:57.142909 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:57.142936 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:57.143355 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:57.143766 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:57.144600 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:57.144604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:57.144616 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:57.144626 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:57.144677 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:57.144696 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:57.144828 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:57.144969 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 12:23:57.145181 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":117,"threads_started":1}
I20260620 12:23:57.148849 21932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8b45ad85f94bb89678a3ead878ab81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42747 } }
I20260620 12:23:57.149133 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:57.149199 21932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:57.149261 21932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8b45ad85f94bb89678a3ead878ab81, State: Initialized, Role: FOLLOWER
I20260620 12:23:57.149660 21932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8b45ad85f94bb89678a3ead878ab81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42747 } }
I20260620 12:23:57.149767 21932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:57.149813 21932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:57.149875 21932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:57.150695 21932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8b45ad85f94bb89678a3ead878ab81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42747 } }
I20260620 12:23:57.150972 21932 leader_election.cc:302] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c8b45ad85f94bb89678a3ead878ab81; no voters:
I20260620 12:23:57.151316 21932 leader_election.cc:288] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:57.151382 21941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:57.151654 21941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [term 1 LEADER]: Becoming Leader. State: Replica: 1c8b45ad85f94bb89678a3ead878ab81, State: Running, Role: LEADER
I20260620 12:23:57.151935 21941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8b45ad85f94bb89678a3ead878ab81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42747 } }
I20260620 12:23:57.152635 21932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:57.153160 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:57.151260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:48009 (local address 127.0.0.1:42747)
0620 12:23:57.151372 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:57.151378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:57.151667 (+ 289us) server_negotiation.cc:407] Connection header received
0620 12:23:57.151836 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:57.151839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:57.151880 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:57.151966 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:57.152708 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:57.152712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:57.152714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:57.152728 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:57.152758 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:57.152770 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:57.152873 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:57.152945 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 12:23:57.152986 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 12:23:57.153139 21941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c8b45ad85f94bb89678a3ead878ab81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8b45ad85f94bb89678a3ead878ab81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42747 } } }
I20260620 12:23:57.153235 21941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:57.153452 21941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8b45ad85f94bb89678a3ead878ab81. Latest consensus state: current_term: 1 leader_uuid: "1c8b45ad85f94bb89678a3ead878ab81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8b45ad85f94bb89678a3ead878ab81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42747 } } }
I20260620 12:23:57.153580 21941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:57.153944 21946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:57.156863 21946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:57.162055 21946 catalog_manager.cc:1347] Generated new cluster ID: afdbb187e6a5468a92e6980cf5dfe657
I20260620 12:23:57.162119 21946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:57.174943 21946 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:57.175275 21946 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:57.183954 21946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c8b45ad85f94bb89678a3ead878ab81: Generated new TSK 0
I20260620 12:23:57.184732 21946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:57.188856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21864
2026-06-20T12:23:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:57 dist-test-slave-9b3m krb5kdc[21963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:57 dist-test-slave-9b3m krb5kdc[21963](info): set up 2 sockets
Jun 20 12:23:57 dist-test-slave-9b3m krb5kdc[21963](info): commencing operation
krb5kdc: starting...
W20260620 12:23:58.225317 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:58 dist-test-slave-9b3m krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:23:58.343849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:58.451159 21976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:58.451459 21976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:58.451550 21976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:58.455652 21976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:58.455747 21976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:58.455787 21976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:58.455845 21976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:58.455886 21976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:58.461194 21976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:58.462255 21976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:58.463277 21976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:58.471769 21984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:58.471843 21982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:58.471778 21981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:58.471988 21976 server_base.cc:1034] running on GCE node
I20260620 12:23:58.472579 21976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:58.473685 21976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:58.474846 21976 hybrid_clock.cc:648] HybridClock initialized: now 1781958238474834 us; error 25 us; skew 500 ppm
Jun 20 12:23:58 dist-test-slave-9b3m krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:58.477571 21976 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:23:58.478391 21976 webserver.cc:460] Webserver started at http://127.0.98.62:33811/ using document root <none> and password file <none>
I20260620 12:23:58.478925 21976 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:58.479015 21976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:58.479261 21976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:58.481303 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "02859384cb674e6982fbed8e18579b3c"
format_stamp: "Formatted at 2026-06-20 12:23:58 on dist-test-slave-9b3m"
server_key: "94674e28404a2d6d2fd9520e4c52ac4f"
server_key_iv: "6d154c440565cbf2a61fdae572937fa9"
server_key_version: "encryptionkey@0"
I20260620 12:23:58.481869 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "02859384cb674e6982fbed8e18579b3c"
format_stamp: "Formatted at 2026-06-20 12:23:58 on dist-test-slave-9b3m"
server_key: "94674e28404a2d6d2fd9520e4c52ac4f"
server_key_iv: "6d154c440565cbf2a61fdae572937fa9"
server_key_version: "encryptionkey@0"
I20260620 12:23:58.485453 21976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:23:58.487757 21991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:58.488903 21976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:23:58.489090 21976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "02859384cb674e6982fbed8e18579b3c"
format_stamp: "Formatted at 2026-06-20 12:23:58 on dist-test-slave-9b3m"
server_key: "94674e28404a2d6d2fd9520e4c52ac4f"
server_key_iv: "6d154c440565cbf2a61fdae572937fa9"
server_key_version: "encryptionkey@0"
I20260620 12:23:58.489212 21976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:58.509241 21976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:58.512482 21976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:58.512740 21976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:58.520874 21976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43697
I20260620 12:23:58.520893 22043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43697 every 8 connection(s)
I20260620 12:23:58.521921 21976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:58.524968 22044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:58.529726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 21976
I20260620 12:23:58.529865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:58.530153 392 external_mini_cluster.cc:1428] Setting key be4d64026a60074705f3782466788665
I20260620 12:23:58.531328 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c: Bootstrap starting.
I20260620 12:23:58.533877 22044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:58.534833 22044 log.cc:824] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:58.537189 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c: No bootstrap required, opened a new log
Jun 20 12:23:58 dist-test-slave-9b3m krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:23:58.541838 22044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02859384cb674e6982fbed8e18579b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } }
I20260620 12:23:58.542079 22044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:58.542145 22044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02859384cb674e6982fbed8e18579b3c, State: Initialized, Role: FOLLOWER
I20260620 12:23:58.542599 22044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02859384cb674e6982fbed8e18579b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } }
I20260620 12:23:58.542762 22044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:58.542860 22044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:58.542965 22044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:58.543893 22044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02859384cb674e6982fbed8e18579b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } }
I20260620 12:23:58.544284 22044 leader_election.cc:302] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02859384cb674e6982fbed8e18579b3c; no voters:
I20260620 12:23:58.544682 22044 leader_election.cc:288] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:58.544764 22049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:58.544976 22049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [term 1 LEADER]: Becoming Leader. State: Replica: 02859384cb674e6982fbed8e18579b3c, State: Running, Role: LEADER
I20260620 12:23:58.545369 22049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02859384cb674e6982fbed8e18579b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } }
I20260620 12:23:58.545785 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:58.531666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34651 (local address 127.0.98.62:43697)
0620 12:23:58.532102 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:58.532117 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:58.532152 (+ 35us) server_negotiation.cc:407] Connection header received
0620 12:23:58.532782 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:58.532800 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:58.533209 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:58.533498 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:58.534548 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:58.535587 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:58.536831 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:58.537438 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:58.537570 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:58.540514 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:58.540519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:58.540547 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:58.540574 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:58.542438 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:58.542989 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:58.542995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:58.543000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:58.543158 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:58.543509 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:58.543512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:58.543514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:58.543903 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:58.544117 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:58.544453 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 12:23:58.544751 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":151,"threads_started":1}
I20260620 12:23:58.546092 22044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:58.547961 22051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02859384cb674e6982fbed8e18579b3c. Latest consensus state: current_term: 1 leader_uuid: "02859384cb674e6982fbed8e18579b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02859384cb674e6982fbed8e18579b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } } }
I20260620 12:23:58.548091 22051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:58.549274 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:58.551394 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02859384cb674e6982fbed8e18579b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02859384cb674e6982fbed8e18579b3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43697 } } }
I20260620 12:23:58.551648 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c [sys.catalog]: This master's current role is: LEADER
W20260620 12:23:58.551412 22063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:23:58.551916 22063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:23:58.552006 22064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:58.552939 22064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:58.559092 22064 catalog_manager.cc:1347] Generated new cluster ID: 572f0da2cc934e618d1ce89068ddfa9d
I20260620 12:23:58.559168 22064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:58.564591 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:58.552413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40517 (local address 127.0.98.62:43697)
0620 12:23:58.552532 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:58.552537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:58.555132 (+ 2595us) server_negotiation.cc:407] Connection header received
0620 12:23:58.555256 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:58.555259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:58.555311 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:58.555408 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:58.556333 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:58.557179 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:58.558328 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:58.558720 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:58.558825 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:58.560863 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:58.560867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:58.560870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:23:58.560896 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:58.562682 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:58.563238 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:58.563241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:58.563243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:58.563327 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:23:58.563673 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:23:58.563677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:23:58.563678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:23:58.563883 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:58.564042 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:58.564333 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 12:23:58.564430 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:23:58.576249 22064 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:58.577638 22064 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:23:58.589615 22064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02859384cb674e6982fbed8e18579b3c: Generated new TSK 0
I20260620 12:23:58.590660 22064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:23:58.599651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 21976
W20260620 12:23:58.605193 21961 connection.cc:535] client connection to 127.0.98.62:43697 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43697): Connection reset by peer (error 104)
2026-06-20T12:23:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:23:58 dist-test-slave-9b3m krb5kdc[22075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:23:58 dist-test-slave-9b3m krb5kdc[22075](info): set up 2 sockets
Jun 20 12:23:58 dist-test-slave-9b3m krb5kdc[22075](info): commencing operation
krb5kdc: starting...
W20260620 12:23:59.637403 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:23:59 dist-test-slave-9b3m krb5kdc[22075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:23:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:23:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:23:59.756320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:23:59.864876 22088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:23:59.865224 22088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:23:59.865314 22088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:23:59.869213 22088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:23:59.869293 22088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:23:59.869329 22088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:23:59.869383 22088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:23:59.869419 22088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:23:59.874630 22088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:23:59.875658 22088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:23:59.876664 22088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:23:59.884752 22096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:59.884801 22093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:23:59.884761 22094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:23:59.885032 22088 server_base.cc:1034] running on GCE node
I20260620 12:23:59.885690 22088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:23:59.886746 22088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:23:59.887929 22088 hybrid_clock.cc:648] HybridClock initialized: now 1781958239887913 us; error 53 us; skew 500 ppm
Jun 20 12:23:59 dist-test-slave-9b3m krb5kdc[22075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:23:59.890690 22088 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:23:59.891820 22088 webserver.cc:460] Webserver started at http://127.0.98.62:36239/ using document root <none> and password file <none>
I20260620 12:23:59.892390 22088 fs_manager.cc:359] Metadata directory not provided
I20260620 12:23:59.892479 22088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:23:59.892712 22088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:23:59.894766 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "39c1fae90ea84a6087e027a47dc23808"
format_stamp: "Formatted at 2026-06-20 12:23:59 on dist-test-slave-9b3m"
server_key: "9289e83911ad83043916c53bf10b6906"
server_key_iv: "37317b92f346845bb2579d6830b0413a"
server_key_version: "encryptionkey@0"
I20260620 12:23:59.895311 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "39c1fae90ea84a6087e027a47dc23808"
format_stamp: "Formatted at 2026-06-20 12:23:59 on dist-test-slave-9b3m"
server_key: "9289e83911ad83043916c53bf10b6906"
server_key_iv: "37317b92f346845bb2579d6830b0413a"
server_key_version: "encryptionkey@0"
I20260620 12:23:59.899052 22088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:23:59.901495 22103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:23:59.902679 22088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:23:59.902818 22088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "39c1fae90ea84a6087e027a47dc23808"
format_stamp: "Formatted at 2026-06-20 12:23:59 on dist-test-slave-9b3m"
server_key: "9289e83911ad83043916c53bf10b6906"
server_key_iv: "37317b92f346845bb2579d6830b0413a"
server_key_version: "encryptionkey@0"
I20260620 12:23:59.902926 22088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:23:59.930171 22088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:23:59.933039 22088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:23:59.933261 22088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:23:59.939982 22088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44381
I20260620 12:23:59.940008 22155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44381 every 8 connection(s)
I20260620 12:23:59.940914 22088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:23:59.942214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 22088
I20260620 12:23:59.942379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:23:59.942682 392 external_mini_cluster.cc:1428] Setting key b8a3c2133b87a92e133cef11db21432c
I20260620 12:23:59.943912 22156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:23:59.950495 22156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808: Bootstrap starting.
I20260620 12:23:59.952762 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:59.944111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47353 (local address 127.0.98.62:44381)
0620 12:23:59.944502 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:59.944521 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:59.944561 (+ 40us) server_negotiation.cc:407] Connection header received
0620 12:23:59.945434 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:59.945456 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:59.945874 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:59.946223 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:59.946620 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:59.948066 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:59.949017 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:59.949752 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:59.949903 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:59.950815 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:59.950821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:59.950835 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:59.951286 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:59.951351 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:59.951380 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:59.951511 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:59.951752 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 12:23:59.951885 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260620 12:23:59.953161 22156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808: Neither blocks nor log segments found. Creating new log.
I20260620 12:23:59.954015 22156 log.cc:824] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808: Log is configured to *not* fsync() on all Append() calls
I20260620 12:23:59.955720 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:23:59.956066 22156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808: No bootstrap required, opened a new log
I20260620 12:23:59.959815 22156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c1fae90ea84a6087e027a47dc23808" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44381 } }
I20260620 12:23:59.960047 22156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:23:59.960124 22156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c1fae90ea84a6087e027a47dc23808, State: Initialized, Role: FOLLOWER
I20260620 12:23:59.960619 22156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c1fae90ea84a6087e027a47dc23808" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44381 } }
I20260620 12:23:59.960785 22156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:23:59.960898 22156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:23:59.961011 22156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:23:59.961613 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:23:59.957704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50021 (local address 127.0.98.62:44381)
0620 12:23:59.957812 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 12:23:59.957817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:23:59.957987 (+ 170us) server_negotiation.cc:407] Connection header received
0620 12:23:59.958133 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:23:59.958136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:23:59.958174 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:23:59.958271 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:23:59.958532 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:59.959027 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:59.959853 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:23:59.960139 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:23:59.960261 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:23:59.961163 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:23:59.961166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:23:59.961171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:23:59.961187 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:23:59.961210 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:23:59.961221 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:23:59.961324 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 12:23:59.961443 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 12:23:59.961477 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:23:59.962098 22156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c1fae90ea84a6087e027a47dc23808" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44381 } }
I20260620 12:23:59.962495 22156 leader_election.cc:302] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c1fae90ea84a6087e027a47dc23808; no voters:
I20260620 12:23:59.962805 22156 leader_election.cc:288] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:23:59.962908 22166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:23:59.963157 22166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [term 1 LEADER]: Becoming Leader. State: Replica: 39c1fae90ea84a6087e027a47dc23808, State: Running, Role: LEADER
I20260620 12:23:59.963495 22166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c1fae90ea84a6087e027a47dc23808" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44381 } }
I20260620 12:23:59.963985 22156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:23:59.965133 22168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c1fae90ea84a6087e027a47dc23808. Latest consensus state: current_term: 1 leader_uuid: "39c1fae90ea84a6087e027a47dc23808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c1fae90ea84a6087e027a47dc23808" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44381 } } }
I20260620 12:23:59.965248 22168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [sys.catalog]: This master's current role is: LEADER
I20260620 12:23:59.965626 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c1fae90ea84a6087e027a47dc23808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c1fae90ea84a6087e027a47dc23808" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44381 } } }
I20260620 12:23:59.965749 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808 [sys.catalog]: This master's current role is: LEADER
W20260620 12:23:59.968438 22181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:23:59.968499 22181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:23:59.968588 22175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:23:59.969316 22175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:23:59.974740 22175 catalog_manager.cc:1347] Generated new cluster ID: 219d9accbf1442d6b6d965768c9a8f83
I20260620 12:23:59.974802 22175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:23:59.993824 22175 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:23:59.994771 22175 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:00.002313 22175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39c1fae90ea84a6087e027a47dc23808: Generated new TSK 0
I20260620 12:24:00.003365 22175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:24:00.017364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 22088
W20260620 12:24:00.022891 22073 connection.cc:535] client connection to 127.0.98.62:44381 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44381): Connection reset by peer (error 104)
2026-06-20T12:24:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:00 dist-test-slave-9b3m krb5kdc[22187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:00 dist-test-slave-9b3m krb5kdc[22187](info): set up 2 sockets
Jun 20 12:24:00 dist-test-slave-9b3m krb5kdc[22187](info): commencing operation
krb5kdc: starting...
W20260620 12:24:01.117313 392 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:01 dist-test-slave-9b3m krb5kdc[22187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:01.240312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:01.348783 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:01.349061 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:01.349102 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:01.352993 22200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:01.353060 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:01.353080 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:01.353127 22200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:01.353159 22200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:01.358280 22200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:01.359241 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:01.360291 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:01.368788 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:01.368752 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:01.368742 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:01.368927 22200 server_base.cc:1034] running on GCE node
I20260620 12:24:01.369547 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:01.370663 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:01.371886 22200 hybrid_clock.cc:648] HybridClock initialized: now 1781958241371864 us; error 56 us; skew 500 ppm
Jun 20 12:24:01 dist-test-slave-9b3m krb5kdc[22187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:01.374814 22200 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:01.375698 22200 webserver.cc:460] Webserver started at http://127.0.98.62:38123/ using document root <none> and password file <none>
I20260620 12:24:01.376212 22200 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:01.376274 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:01.376448 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:01.378501 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "2caf7b7475c54311a6b82481ca1d7792"
format_stamp: "Formatted at 2026-06-20 12:24:01 on dist-test-slave-9b3m"
server_key: "a22cbf4d33a4d8b059e74687acd62cdd"
server_key_iv: "7b78f6c4b0d355505762fbc35f0ffbf9"
server_key_version: "encryptionkey@0"
I20260620 12:24:01.379068 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "2caf7b7475c54311a6b82481ca1d7792"
format_stamp: "Formatted at 2026-06-20 12:24:01 on dist-test-slave-9b3m"
server_key: "a22cbf4d33a4d8b059e74687acd62cdd"
server_key_iv: "7b78f6c4b0d355505762fbc35f0ffbf9"
server_key_version: "encryptionkey@0"
I20260620 12:24:01.382845 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:24:01.385275 22215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:01.386492 22200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:24:01.386634 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "2caf7b7475c54311a6b82481ca1d7792"
format_stamp: "Formatted at 2026-06-20 12:24:01 on dist-test-slave-9b3m"
server_key: "a22cbf4d33a4d8b059e74687acd62cdd"
server_key_iv: "7b78f6c4b0d355505762fbc35f0ffbf9"
server_key_version: "encryptionkey@0"
I20260620 12:24:01.386742 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:01.391366 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:01.394200 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:01.394392 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:01.401664 22200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43189
I20260620 12:24:01.401687 22267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43189 every 8 connection(s)
I20260620 12:24:01.402586 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:01.405566 22268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:01.406142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 22200
I20260620 12:24:01.406315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:01.406620 392 external_mini_cluster.cc:1428] Setting key 88069567198ef29a73cd6cad86fc06f7
I20260620 12:24:01.411815 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:01.408028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38131 (local address 127.0.98.62:43189)
0620 12:24:01.408466 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:01.408481 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:01.408514 (+ 33us) server_negotiation.cc:407] Connection header received
0620 12:24:01.409293 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:01.409317 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:01.409586 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:01.409913 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:01.410728 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:01.410733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:01.410743 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:24:01.410754 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:01.410806 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:24:01.410825 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:01.410963 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:01.411031 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 12:24:01.411185 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
I20260620 12:24:01.412819 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792: Bootstrap starting.
I20260620 12:24:01.414489 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:24:01.415330 22268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:01.416220 22268 log.cc:824] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:01.418109 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792: No bootstrap required, opened a new log
I20260620 12:24:01.418344 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:01.416431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55459 (local address 127.0.98.62:43189)
0620 12:24:01.416569 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:01.416576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:01.416865 (+ 289us) server_negotiation.cc:407] Connection header received
0620 12:24:01.417023 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:01.417027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:01.417082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:01.417179 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:01.417925 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:01.417929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:01.417932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:24:01.417944 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:01.417970 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:24:01.417988 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:01.418080 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:01.418174 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 12:24:01.418213 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 12:24:01.421818 22268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf7b7475c54311a6b82481ca1d7792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } }
I20260620 12:24:01.422046 22268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:01.422115 22268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caf7b7475c54311a6b82481ca1d7792, State: Initialized, Role: FOLLOWER
I20260620 12:24:01.422566 22268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf7b7475c54311a6b82481ca1d7792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } }
I20260620 12:24:01.422715 22268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:24:01.422783 22268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:24:01.422879 22268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:01.423890 22268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf7b7475c54311a6b82481ca1d7792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } }
I20260620 12:24:01.424221 22268 leader_election.cc:302] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2caf7b7475c54311a6b82481ca1d7792; no voters:
I20260620 12:24:01.424532 22268 leader_election.cc:288] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:24:01.424777 22278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:01.424988 22278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [term 1 LEADER]: Becoming Leader. State: Replica: 2caf7b7475c54311a6b82481ca1d7792, State: Running, Role: LEADER
I20260620 12:24:01.425354 22278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf7b7475c54311a6b82481ca1d7792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } }
I20260620 12:24:01.425783 22268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:01.427006 22280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2caf7b7475c54311a6b82481ca1d7792. Latest consensus state: current_term: 1 leader_uuid: "2caf7b7475c54311a6b82481ca1d7792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf7b7475c54311a6b82481ca1d7792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } }
I20260620 12:24:01.427045 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2caf7b7475c54311a6b82481ca1d7792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf7b7475c54311a6b82481ca1d7792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } }
I20260620 12:24:01.427199 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:01.427501 22280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792 [sys.catalog]: This master's current role is: LEADER
W20260620 12:24:01.430737 22293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:24:01.430814 22293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:24:01.430910 22290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:01.431608 22290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:01.436520 22290 catalog_manager.cc:1347] Generated new cluster ID: 3fff5755252d4f9198a5e9b56f8aa48d
I20260620 12:24:01.436582 22290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:01.459641 22290 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:01.459980 22290 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:01.468170 22290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2caf7b7475c54311a6b82481ca1d7792: Generated new TSK 0
I20260620 12:24:01.468848 22290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:24:01.479074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 22200
2026-06-20T12:24:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:01 dist-test-slave-9b3m krb5kdc[22299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:01 dist-test-slave-9b3m krb5kdc[22299](info): set up 2 sockets
Jun 20 12:24:01 dist-test-slave-9b3m krb5kdc[22299](info): commencing operation
krb5kdc: starting...
W20260620 12:24:02.545258 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:02 dist-test-slave-9b3m krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:02.666164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36133
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:02.773882 22312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:02.774166 22312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:02.774207 22312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:02.778199 22312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:02.778257 22312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:02.778276 22312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:02.778293 22312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:02.778308 22312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:02.783409 22312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:02.784348 22312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:02.785277 22312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:02.793601 22317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:02.793772 22318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:02.793721 22312 server_base.cc:1034] running on GCE node
W20260620 12:24:02.793610 22320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:02.794471 22312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:02.795564 22312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:02.796775 22312 hybrid_clock.cc:648] HybridClock initialized: now 1781958242796746 us; error 39 us; skew 500 ppm
Jun 20 12:24:02 dist-test-slave-9b3m krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:02.802852 22312 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:02.803774 22312 webserver.cc:460] Webserver started at http://127.0.98.62:44989/ using document root <none> and password file <none>
I20260620 12:24:02.804317 22312 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:02.804404 22312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:02.804634 22312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:02.806663 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "bc62bb7802034560be59f24b4840a128"
format_stamp: "Formatted at 2026-06-20 12:24:02 on dist-test-slave-9b3m"
server_key: "23ca73101376c99e1f63f7596cf372f2"
server_key_iv: "bfddd0a90825dc58bd682343847b2a38"
server_key_version: "encryptionkey@0"
I20260620 12:24:02.807209 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc62bb7802034560be59f24b4840a128"
format_stamp: "Formatted at 2026-06-20 12:24:02 on dist-test-slave-9b3m"
server_key: "23ca73101376c99e1f63f7596cf372f2"
server_key_iv: "bfddd0a90825dc58bd682343847b2a38"
server_key_version: "encryptionkey@0"
I20260620 12:24:02.810941 22312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:24:02.813331 22327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:02.814502 22312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:24:02.814649 22312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "bc62bb7802034560be59f24b4840a128"
format_stamp: "Formatted at 2026-06-20 12:24:02 on dist-test-slave-9b3m"
server_key: "23ca73101376c99e1f63f7596cf372f2"
server_key_iv: "bfddd0a90825dc58bd682343847b2a38"
server_key_version: "encryptionkey@0"
I20260620 12:24:02.814761 22312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:02.840482 22312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:02.843781 22312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:02.843983 22312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:02.851356 22312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36133
I20260620 12:24:02.851384 22379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36133 every 8 connection(s)
I20260620 12:24:02.852275 22312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:02.854984 22380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:02.860423 22380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128: Bootstrap starting.
I20260620 12:24:02.862030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 22312
I20260620 12:24:02.862175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:02.862485 392 external_mini_cluster.cc:1428] Setting key 09e0593a395ce3b43549dd7346d958d8
I20260620 12:24:02.863126 22380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:02.863996 22380 log.cc:824] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:02.865978 22380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128: No bootstrap required, opened a new log
I20260620 12:24:02.869370 22380 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc62bb7802034560be59f24b4840a128" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } }
I20260620 12:24:02.869601 22380 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:02.869696 22380 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc62bb7802034560be59f24b4840a128, State: Initialized, Role: FOLLOWER
I20260620 12:24:02.870208 22380 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc62bb7802034560be59f24b4840a128" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } }
I20260620 12:24:02.870385 22380 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:24:02.870465 22380 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:24:02.870563 22380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 12:24:02 dist-test-slave-9b3m krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:02.871479 22380 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc62bb7802034560be59f24b4840a128" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } }
I20260620 12:24:02.871798 22380 leader_election.cc:302] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc62bb7802034560be59f24b4840a128; no voters:
I20260620 12:24:02.872130 22380 leader_election.cc:288] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:24:02.872195 22385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:02.872400 22385 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [term 1 LEADER]: Becoming Leader. State: Replica: bc62bb7802034560be59f24b4840a128, State: Running, Role: LEADER
I20260620 12:24:02.872752 22385 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc62bb7802034560be59f24b4840a128" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } }
I20260620 12:24:02.873426 22380 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:02.873962 22385 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc62bb7802034560be59f24b4840a128. Latest consensus state: current_term: 1 leader_uuid: "bc62bb7802034560be59f24b4840a128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc62bb7802034560be59f24b4840a128" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } } }
I20260620 12:24:02.874130 22385 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:02.874403 22386 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc62bb7802034560be59f24b4840a128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc62bb7802034560be59f24b4840a128" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } } }
I20260620 12:24:02.874511 22386 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:02.874827 22392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:02.877714 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:02.863592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49225 (local address 127.0.98.62:36133)
0620 12:24:02.864128 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:02.864148 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:02.864189 (+ 41us) server_negotiation.cc:407] Connection header received
0620 12:24:02.865008 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:02.865027 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:02.865432 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:02.865781 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:02.866576 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:02.867552 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:02.868415 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:02.868880 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:02.869102 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:02.871949 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:02.871955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:02.871971 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:02.872002 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:02.874169 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:02.874735 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:02.874740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:02.874745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:02.874886 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:02.875223 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:02.875226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:02.875228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:02.875727 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:02.875924 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:02.876249 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 12:24:02.876742 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":141,"threads_started":1}
I20260620 12:24:02.878309 22392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:02.880546 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:24:02.884495 22392 catalog_manager.cc:1347] Generated new cluster ID: 57e4a3546ed146b59f3a9f136ac4f3d5
I20260620 12:24:02.884613 22392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:02.891587 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:02.882517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39927 (local address 127.0.98.62:36133)
0620 12:24:02.882618 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:02.882622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:02.882913 (+ 291us) server_negotiation.cc:407] Connection header received
0620 12:24:02.883035 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:02.883041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:02.883099 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:02.883202 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:02.883848 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:02.884977 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:02.886121 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:02.886458 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:02.886531 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:02.888150 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:02.888154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:02.888157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:02.888179 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:02.889763 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:02.890201 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:02.890205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:02.890206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:02.890277 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:02.890511 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:02.890514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:02.890516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:02.890692 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:02.890797 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:02.891333 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 12:24:02.891435 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 12:24:02.894861 22392 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:02.895998 22392 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:02.902667 22392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc62bb7802034560be59f24b4840a128: Generated new TSK 0
I20260620 12:24:02.903437 22392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:24:02.909775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 22312
2026-06-20T12:24:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:02 dist-test-slave-9b3m krb5kdc[22411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:02 dist-test-slave-9b3m krb5kdc[22411](info): set up 2 sockets
Jun 20 12:24:02 dist-test-slave-9b3m krb5kdc[22411](info): commencing operation
krb5kdc: starting...
W20260620 12:24:03.949585 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:04 dist-test-slave-9b3m krb5kdc[22411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:04.072446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:04.180160 22424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:04.180395 22424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:04.180436 22424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:04.184376 22424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:04.184429 22424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:04.184448 22424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:04.184464 22424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:04.184479 22424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:04.189555 22424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:04.190436 22424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:04.191329 22424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:04.199478 22432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:04.199483 22429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:04.199698 22424 server_base.cc:1034] running on GCE node
W20260620 12:24:04.199505 22430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:04.200366 22424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:04.201587 22424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:04.202801 22424 hybrid_clock.cc:648] HybridClock initialized: now 1781958244202776 us; error 38 us; skew 500 ppm
Jun 20 12:24:04 dist-test-slave-9b3m krb5kdc[22411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:04.205885 22424 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:04.206733 22424 webserver.cc:460] Webserver started at http://127.0.98.62:33075/ using document root <none> and password file <none>
I20260620 12:24:04.207288 22424 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:04.207379 22424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:04.207612 22424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:04.209683 22424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "a0150e3fa3a846f2861d36857da057e4"
format_stamp: "Formatted at 2026-06-20 12:24:04 on dist-test-slave-9b3m"
server_key: "4c3aaa9e7033f355aba6c42bc998913d"
server_key_iv: "30e8ba2d1b81c8ac1a4b579abb766913"
server_key_version: "encryptionkey@0"
I20260620 12:24:04.210253 22424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0150e3fa3a846f2861d36857da057e4"
format_stamp: "Formatted at 2026-06-20 12:24:04 on dist-test-slave-9b3m"
server_key: "4c3aaa9e7033f355aba6c42bc998913d"
server_key_iv: "30e8ba2d1b81c8ac1a4b579abb766913"
server_key_version: "encryptionkey@0"
I20260620 12:24:04.213886 22424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 12:24:04.216219 22439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:04.217478 22424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 12:24:04.217612 22424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "a0150e3fa3a846f2861d36857da057e4"
format_stamp: "Formatted at 2026-06-20 12:24:04 on dist-test-slave-9b3m"
server_key: "4c3aaa9e7033f355aba6c42bc998913d"
server_key_iv: "30e8ba2d1b81c8ac1a4b579abb766913"
server_key_version: "encryptionkey@0"
I20260620 12:24:04.217722 22424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:04.245733 22424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:04.248805 22424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:04.249022 22424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:04.256212 22424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46485
I20260620 12:24:04.256233 22491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46485 every 8 connection(s)
I20260620 12:24:04.257272 22424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:04.258420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 22424
I20260620 12:24:04.258574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:04.258934 392 external_mini_cluster.cc:1428] Setting key 661080b45a19d97f818cee01e3b2bb17
I20260620 12:24:04.260586 22492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:04.266111 22492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4: Bootstrap starting.
I20260620 12:24:04.267903 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:04.260373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40675 (local address 127.0.98.62:46485)
0620 12:24:04.260847 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:04.260862 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:04.260898 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:24:04.261595 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:04.261611 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:04.261942 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:04.262269 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:04.262667 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:04.263638 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:04.264542 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:04.265009 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:04.265224 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:04.266111 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:04.266116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:04.266128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:24:04.266559 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:04.266611 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:24:04.266634 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:04.266747 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:04.266897 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 12:24:04.267104 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
I20260620 12:24:04.268461 22492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:04.269330 22492 log.cc:824] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:04.270907 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:24:04.271435 22492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4: No bootstrap required, opened a new log
I20260620 12:24:04.274981 22492 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0150e3fa3a846f2861d36857da057e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260620 12:24:04.275264 22492 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:04.275326 22492 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0150e3fa3a846f2861d36857da057e4, State: Initialized, Role: FOLLOWER
I20260620 12:24:04.275777 22492 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0150e3fa3a846f2861d36857da057e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260620 12:24:04.275929 22492 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:24:04.275998 22492 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:24:04.276116 22492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:04.277406 22492 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0150e3fa3a846f2861d36857da057e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260620 12:24:04.277667 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:04.272920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53409 (local address 127.0.98.62:46485)
0620 12:24:04.273034 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:04.273039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:04.273213 (+ 174us) server_negotiation.cc:407] Connection header received
0620 12:24:04.273349 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:04.273352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:04.273389 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:04.273446 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:04.273820 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:04.274703 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:04.275789 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:04.276184 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:04.276339 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:04.277153 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:04.277156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:04.277158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:24:04.277169 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:04.277192 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:24:04.277202 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:04.277302 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:04.277458 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 12:24:04.277531 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 12:24:04.277776 22492 leader_election.cc:302] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0150e3fa3a846f2861d36857da057e4; no voters:
I20260620 12:24:04.278156 22492 leader_election.cc:288] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:24:04.278294 22502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:04.278496 22502 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [term 1 LEADER]: Becoming Leader. State: Replica: a0150e3fa3a846f2861d36857da057e4, State: Running, Role: LEADER
I20260620 12:24:04.278865 22502 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0150e3fa3a846f2861d36857da057e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } }
I20260620 12:24:04.279424 22492 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:04.280484 22503 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0150e3fa3a846f2861d36857da057e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0150e3fa3a846f2861d36857da057e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } } }
I20260620 12:24:04.280611 22503 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:04.280910 22504 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0150e3fa3a846f2861d36857da057e4. Latest consensus state: current_term: 1 leader_uuid: "a0150e3fa3a846f2861d36857da057e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0150e3fa3a846f2861d36857da057e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46485 } } }
I20260620 12:24:04.280997 22504 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:04.280982 22511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:04.284175 22511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:04.289565 22511 catalog_manager.cc:1347] Generated new cluster ID: d27712a130de46dfb69aeb3275b63649
I20260620 12:24:04.289625 22511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:04.298602 22511 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:04.299744 22511 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:04.308327 22511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0150e3fa3a846f2861d36857da057e4: Generated new TSK 0
I20260620 12:24:04.309139 22511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:24:04.318364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 22424
2026-06-20T12:24:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:04 dist-test-slave-9b3m krb5kdc[22523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:04 dist-test-slave-9b3m krb5kdc[22523](info): set up 2 sockets
Jun 20 12:24:04 dist-test-slave-9b3m krb5kdc[22523](info): commencing operation
krb5kdc: starting...
W20260620 12:24:05.401377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:05 dist-test-slave-9b3m krb5kdc[22523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:05.524063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34811
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:05.632541 22536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:05.632819 22536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:05.632891 22536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:05.636812 22536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:05.636904 22536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:05.636942 22536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:05.636981 22536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:05.637022 22536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:05.642360 22536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:05.643399 22536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:05.644465 22536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:05.653096 22542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:05.653235 22536 server_base.cc:1034] running on GCE node
W20260620 12:24:05.653101 22541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:05.653136 22544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:05.653980 22536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:05.655093 22536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:05.656286 22536 hybrid_clock.cc:648] HybridClock initialized: now 1781958245656267 us; error 31 us; skew 500 ppm
Jun 20 12:24:05 dist-test-slave-9b3m krb5kdc[22523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:05.659170 22536 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:05.660095 22536 webserver.cc:460] Webserver started at http://127.0.98.62:45001/ using document root <none> and password file <none>
I20260620 12:24:05.660657 22536 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:05.660748 22536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:05.660956 22536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:05.662977 22536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "04b92c1d311a4982bfd4857113562c6b"
format_stamp: "Formatted at 2026-06-20 12:24:05 on dist-test-slave-9b3m"
server_key: "e82f2f6b5c4009e59abd5fe91598e2da"
server_key_iv: "15c54eec5e71d406cf76f7e5f35eb068"
server_key_version: "encryptionkey@0"
I20260620 12:24:05.663522 22536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "04b92c1d311a4982bfd4857113562c6b"
format_stamp: "Formatted at 2026-06-20 12:24:05 on dist-test-slave-9b3m"
server_key: "e82f2f6b5c4009e59abd5fe91598e2da"
server_key_iv: "15c54eec5e71d406cf76f7e5f35eb068"
server_key_version: "encryptionkey@0"
I20260620 12:24:05.667207 22536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:24:05.669687 22551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:05.670913 22536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:24:05.671100 22536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "04b92c1d311a4982bfd4857113562c6b"
format_stamp: "Formatted at 2026-06-20 12:24:05 on dist-test-slave-9b3m"
server_key: "e82f2f6b5c4009e59abd5fe91598e2da"
server_key_iv: "15c54eec5e71d406cf76f7e5f35eb068"
server_key_version: "encryptionkey@0"
I20260620 12:24:05.671249 22536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:05.675674 22536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:05.678989 22536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:05.679179 22536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:05.686408 22536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34811
I20260620 12:24:05.686478 22603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34811 every 8 connection(s)
I20260620 12:24:05.687318 22536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:05.689563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 22536
I20260620 12:24:05.689702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:05.689999 392 external_mini_cluster.cc:1428] Setting key c2050541766a23cfb09775c33fb2c8f0
I20260620 12:24:05.690070 22604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:05.695317 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:05.691334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56105 (local address 127.0.98.62:34811)
0620 12:24:05.691759 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:05.691774 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:05.691808 (+ 34us) server_negotiation.cc:407] Connection header received
0620 12:24:05.692697 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:05.692715 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:05.692998 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:05.693377 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:05.694195 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:05.694199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:05.694210 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:24:05.694220 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:05.694270 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:24:05.694288 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:05.694438 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:05.694508 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 12:24:05.694653 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
I20260620 12:24:05.696493 22604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b: Bootstrap starting.
I20260620 12:24:05.697983 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:24:05.698823 22604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:05.699640 22604 log.cc:824] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:05.701486 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:05.699868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57987 (local address 127.0.98.62:34811)
0620 12:24:05.699986 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:05.699989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:05.700238 (+ 249us) server_negotiation.cc:407] Connection header received
0620 12:24:05.700317 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:05.700322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:05.700359 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:05.700449 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:05.701151 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:05.701154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:05.701156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 12:24:05.701166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:05.701188 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 12:24:05.701198 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:05.701263 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:05.701372 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 12:24:05.701401 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:24:05.701929 22604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b: No bootstrap required, opened a new log
I20260620 12:24:05.705384 22604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b92c1d311a4982bfd4857113562c6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34811 } }
I20260620 12:24:05.705614 22604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 12:24:05.705701 22604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b92c1d311a4982bfd4857113562c6b, State: Initialized, Role: FOLLOWER
I20260620 12:24:05.706157 22604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b92c1d311a4982bfd4857113562c6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34811 } }
I20260620 12:24:05.706305 22604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 12:24:05.706391 22604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 12:24:05.706486 22604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:05.707329 22604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b92c1d311a4982bfd4857113562c6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34811 } }
I20260620 12:24:05.707602 22604 leader_election.cc:302] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04b92c1d311a4982bfd4857113562c6b; no voters:
I20260620 12:24:05.707973 22604 leader_election.cc:288] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 12:24:05.708001 22614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:05.708191 22614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [term 1 LEADER]: Becoming Leader. State: Replica: 04b92c1d311a4982bfd4857113562c6b, State: Running, Role: LEADER
I20260620 12:24:05.708535 22614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b92c1d311a4982bfd4857113562c6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34811 } }
I20260620 12:24:05.709180 22604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:05.710253 22616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04b92c1d311a4982bfd4857113562c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b92c1d311a4982bfd4857113562c6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34811 } } }
I20260620 12:24:05.710280 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04b92c1d311a4982bfd4857113562c6b. Latest consensus state: current_term: 1 leader_uuid: "04b92c1d311a4982bfd4857113562c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b92c1d311a4982bfd4857113562c6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34811 } } }
I20260620 12:24:05.710448 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:05.710716 22616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:05.710858 22623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:05.713982 22623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:05.719518 22623 catalog_manager.cc:1347] Generated new cluster ID: 0892a20d8c2e410086e86d8657da04af
I20260620 12:24:05.719573 22623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:05.730082 22623 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:05.730376 22623 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:05.736200 22623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04b92c1d311a4982bfd4857113562c6b: Generated new TSK 0
I20260620 12:24:05.736857 22623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 12:24:05.745074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 22536
2026-06-20T12:24:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1426 ms)
[----------] 18 tests from AuthTokenIssuingTest (25552 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:05 dist-test-slave-9b3m krb5kdc[22635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:05 dist-test-slave-9b3m krb5kdc[22635](info): set up 2 sockets
Jun 20 12:24:05 dist-test-slave-9b3m krb5kdc[22635](info): commencing operation
krb5kdc: starting...
W20260620 12:24:06.777495 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:06 dist-test-slave-9b3m krb5kdc[22635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:06.896816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33917,127.0.98.61:37099,127.0.98.60:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:07.005810 22648 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:24:07.006093 22648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:07.006117 22648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:07.006212 22648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:07.010224 22648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:07.010277 22648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:07.010296 22648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:07.010313 22648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:07.010357 22648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:07.015604 22648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33917,127.0.98.61:37099,127.0.98.60:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:07.016629 22648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:07.017648 22648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:07.026006 22654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:07.026016 22656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:07.026046 22653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:07.026226 22648 server_base.cc:1034] running on GCE node
I20260620 12:24:07.026823 22648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:07.027949 22648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:07.029135 22648 hybrid_clock.cc:648] HybridClock initialized: now 1781958247029107 us; error 41 us; skew 500 ppm
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:07.031891 22648 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:07.032801 22648 webserver.cc:460] Webserver started at http://127.0.98.62:40421/ using document root <none> and password file <none>
I20260620 12:24:07.033428 22648 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:07.033524 22648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:07.033757 22648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:07.035778 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "e0f83c1d5d74463b89e9ad1a210e9220"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "b3933e134ad06433e5a966c998cea0e4"
server_key_iv: "def14df92418d51dec67e0ff55888c5e"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.036439 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0f83c1d5d74463b89e9ad1a210e9220"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "b3933e134ad06433e5a966c998cea0e4"
server_key_iv: "def14df92418d51dec67e0ff55888c5e"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.040194 22648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 12:24:07.042779 22663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:07.043998 22648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 12:24:07.044139 22648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "e0f83c1d5d74463b89e9ad1a210e9220"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "b3933e134ad06433e5a966c998cea0e4"
server_key_iv: "def14df92418d51dec67e0ff55888c5e"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.044261 22648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:07.057849 22648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:07.060926 22648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:07.061215 22648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:07.068267 22648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33917
I20260620 12:24:07.068580 22715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33917 every 8 connection(s)
I20260620 12:24:07.069258 22648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:07.072149 22716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:07.072716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 22648
I20260620 12:24:07.072850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:07.073231 392 external_mini_cluster.cc:1428] Setting key 99b9143960fa4e19cf834ce3b2e48ace
I20260620 12:24:07.076990 22716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:07.093086 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.079369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33917 (local address 127.0.0.1:38012)
0620 12:24:07.079854 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.079908 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.080921 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.081718 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.081723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.082172 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.082438 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.082452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.083882 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.083892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.084681 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.084690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.085542 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.085550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.085707 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.086367 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.086395 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.088220 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.090234 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.090244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.090253 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.090476 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.090804 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.090807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.090809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.090917 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.091511 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.091517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.091685 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.091884 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.092335 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":138,"thread_start_us":154,"threads_started":1}
I20260620 12:24:07.093093 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.079370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.0.98.62:33917)
0620 12:24:07.079928 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.079941 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.080045 (+ 104us) server_negotiation.cc:407] Connection header received
0620 12:24:07.081316 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.081321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.081506 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.081714 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.082594 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.083742 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.084882 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.085366 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.085498 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.088372 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.088378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.088387 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.088406 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.090095 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.090587 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.090592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.090594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.090701 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.091019 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.091022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.091024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.091390 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.091529 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.091942 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.092224 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":436,"spinlock_wait_cycles":46976,"thread_start_us":158,"threads_started":1}
I20260620 12:24:07.095460 22716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:07.096231 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.095932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:54074)
0620 12:24:07.096108 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.096176 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 12:24:07.096421 22666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
I20260620 12:24:07.097333 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.097031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:54078)
0620 12:24:07.097241 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.097284 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260620 12:24:07.097463 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37099: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 12:24:07.100690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37099
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33917,127.0.98.61:37099,127.0.98.60:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:24:07.146762 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } attempt: 1
I20260620 12:24:07.147691 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.147401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:54082)
0620 12:24:07.147536 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.147610 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 12:24:07.148514 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.148238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:54092)
0620 12:24:07.148343 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.148418 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 12:24:07.148655 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37099: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
W20260620 12:24:07.208390 22723 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:24:07.208634 22723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:07.208657 22723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:07.208693 22723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:07.212616 22723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:07.212664 22723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:07.212683 22723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:07.212699 22723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:07.212749 22723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:07.216806 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } attempt: 2
I20260620 12:24:07.217723 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.217379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:54098)
0620 12:24:07.217557 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.217629 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 12:24:07.218117 22723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33917,127.0.98.61:37099,127.0.98.60:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:07.218469 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.218215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:54112)
0620 12:24:07.218342 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.218406 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 12:24:07.218626 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37099: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37099: connect: Connection refused (error 111)
I20260620 12:24:07.219183 22723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:07.220110 22723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:07.228291 22728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:07.228370 22729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:07.228370 22731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:07.228444 22723 server_base.cc:1034] running on GCE node
I20260620 12:24:07.229038 22723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:07.230152 22723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:07.231338 22723 hybrid_clock.cc:648] HybridClock initialized: now 1781958247231313 us; error 48 us; skew 500 ppm
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:07.234009 22723 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 12:24:07.234876 22723 webserver.cc:460] Webserver started at http://127.0.98.61:44361/ using document root <none> and password file <none>
I20260620 12:24:07.235402 22723 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:07.235491 22723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:07.235734 22723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:07.237718 22723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/data/instance:
uuid: "2f53592f58164002b8892bda7d61eeff"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "4be4ae39732a6a9500a602760e015cdc"
server_key_iv: "d9f099741198072dd5ed492dc26ff3b4"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.238193 22723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/wal/instance:
uuid: "2f53592f58164002b8892bda7d61eeff"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "4be4ae39732a6a9500a602760e015cdc"
server_key_iv: "d9f099741198072dd5ed492dc26ff3b4"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.241638 22723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:24:07.243876 22738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:07.245245 22723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:24:07.245374 22723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/wal
uuid: "2f53592f58164002b8892bda7d61eeff"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "4be4ae39732a6a9500a602760e015cdc"
server_key_iv: "d9f099741198072dd5ed492dc26ff3b4"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.245481 22723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:07.256891 22723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:07.259677 22723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:07.259881 22723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:07.266907 22723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37099
I20260620 12:24:07.267185 22790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37099 every 8 connection(s)
I20260620 12:24:07.267829 22723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:24:07.270514 22791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:07.273996 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:07.276477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 22723
I20260620 12:24:07.276623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-1/wal/instance
I20260620 12:24:07.276949 392 external_mini_cluster.cc:1428] Setting key 61ce8413590040bf2a8c285c242b76f6
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 12:24:07.288301 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.275850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.0.98.62:33917)
0620 12:24:07.275985 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.275989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.276869 (+ 880us) server_negotiation.cc:407] Connection header received
0620 12:24:07.277814 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.277820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.277887 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.277997 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.279089 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.279844 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.281707 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.281936 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.282022 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.284732 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.284735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.284737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.284759 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.286216 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.286785 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.286790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.286791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.286871 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.287239 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.287242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.287244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.287418 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.287527 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.288035 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.288180 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:24:07.289332 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.276185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33917 (local address 127.0.0.1:38022)
0620 12:24:07.276712 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.276762 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.277605 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.278142 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.278148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.278456 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.278938 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.278950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.279990 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.279996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.281582 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.281594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.282101 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.282108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.282344 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.282971 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.282997 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.284560 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.286340 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.286345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.286358 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.286681 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.286973 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.286976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.286978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.287112 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.287525 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.287532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.287773 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.287961 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.288260 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":143,"threads_started":1}
I20260620 12:24:07.290894 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:24:07.299556 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.291346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:54128)
0620 12:24:07.291546 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.291564 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.291651 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.292069 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.292071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.292253 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.292324 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.292329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.293333 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.293336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.293865 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.293872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.294427 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.294430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.294569 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.295054 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.295065 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.296209 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.298073 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.298076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.298078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.298261 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.298543 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.298546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.298548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.298621 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.299056 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.299059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.299146 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.299254 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.299374 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":39}
I20260620 12:24:07.299666 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.291457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.0.98.61:37099)
0620 12:24:07.291755 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.291764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.291782 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:24:07.291815 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.291819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.291952 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.292089 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.292429 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.293226 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.293974 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.294316 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.294430 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.296335 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.296344 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.296363 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.296383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.297954 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.298365 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.298373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.298374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.298446 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.298726 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.298731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.298732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.298946 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.299080 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.299347 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.299462 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":194,"thread_start_us":128,"threads_started":1}
I20260620 12:24:07.301440 22791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:07.302086 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.301832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50776)
0620 12:24:07.301952 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.302018 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 12:24:07.302299 22739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
I20260620 12:24:07.303285 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.302933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50788)
0620 12:24:07.303181 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.303230 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
W20260620 12:24:07.303480 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 12:24:07.304525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38619
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33917,127.0.98.61:37099,127.0.98.60:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:24:07.309805 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } attempt: 3
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:24:07.319258 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.310271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:54142)
0620 12:24:07.310375 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.310391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.310473 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.310780 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.310782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.310941 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.311043 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.311048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.312179 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.312182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.312776 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.312785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.313525 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.313528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.313618 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.314152 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.314165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.316046 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.317954 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.317957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.317958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.318147 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.318434 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.318436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.318438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.318511 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.318889 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.318891 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.318967 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.319046 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.319143 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 12:24:07.319367 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.310351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.0.98.61:37099)
0620 12:24:07.310487 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.310492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.310507 (+ 15us) server_negotiation.cc:407] Connection header received
0620 12:24:07.310553 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.310558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.310617 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.310720 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.311163 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.312048 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.312892 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.313372 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.313500 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.316187 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.316191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.316194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.316219 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.317845 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.318262 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.318265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.318267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.318348 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.318594 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.318597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.318598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.318803 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.318894 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.319112 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.319222 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:24:07.320652 22716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:07.321357 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.321137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50798)
0620 12:24:07.321239 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.321289 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 12:24:07.322098 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.321877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50804)
0620 12:24:07.322012 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.322052 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 12:24:07.322290 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
I20260620 12:24:07.352753 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } attempt: 1
I20260620 12:24:07.353462 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } attempt: 1
I20260620 12:24:07.353647 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.353286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50806)
0620 12:24:07.353460 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.353523 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:24:07.354104 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.353834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50810)
0620 12:24:07.353972 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.354029 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 12:24:07.354491 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.354251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50824)
0620 12:24:07.354361 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.354403 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 12:24:07.354687 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
I20260620 12:24:07.354991 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.354773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50828)
0620 12:24:07.354869 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.354945 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 12:24:07.355127 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
W20260620 12:24:07.416097 22798 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:24:07.416364 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:07.416388 22798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:07.416424 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:07.420537 22798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:07.420593 22798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:07.420612 22798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:07.420629 22798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:07.420643 22798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:07.422857 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } attempt: 2
I20260620 12:24:07.423672 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.423341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50844)
0620 12:24:07.423505 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.423571 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:24:07.424479 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.424200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50860)
0620 12:24:07.424352 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.424413 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 12:24:07.424698 22791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
I20260620 12:24:07.426573 22798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33917,127.0.98.61:37099,127.0.98.60:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:07.427572 22798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:07.428537 22798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:24:07.430305 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } attempt: 2
I20260620 12:24:07.430959 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.430720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50866)
0620 12:24:07.430839 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.430897 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 12:24:07.431712 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.431477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:50872)
0620 12:24:07.431616 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.431661 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 12:24:07.431876 22716 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38619: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38619: connect: Connection refused (error 111)
W20260620 12:24:07.437264 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:07.437314 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:07.437240 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:07.437364 22798 server_base.cc:1034] running on GCE node
I20260620 12:24:07.438035 22798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:07.439131 22798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:07.440323 22798 hybrid_clock.cc:648] HybridClock initialized: now 1781958247440308 us; error 27 us; skew 500 ppm
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:07.443076 22798 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 12:24:07.443962 22798 webserver.cc:460] Webserver started at http://127.0.98.60:45271/ using document root <none> and password file <none>
I20260620 12:24:07.444505 22798 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:07.444597 22798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:07.444844 22798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:07.446899 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/data/instance:
uuid: "880892c0f07e4b2788d1a6985803a17e"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "41ccf3c2702e751312521ba45840a242"
server_key_iv: "26a1163f3eed2f033f3f70e1db9f46f5"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.447515 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/wal/instance:
uuid: "880892c0f07e4b2788d1a6985803a17e"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "41ccf3c2702e751312521ba45840a242"
server_key_iv: "26a1163f3eed2f033f3f70e1db9f46f5"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.451386 22798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 12:24:07.453807 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:07.454931 22798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 12:24:07.455067 22798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/wal
uuid: "880892c0f07e4b2788d1a6985803a17e"
format_stamp: "Formatted at 2026-06-20 12:24:07 on dist-test-slave-9b3m"
server_key: "41ccf3c2702e751312521ba45840a242"
server_key_iv: "26a1163f3eed2f033f3f70e1db9f46f5"
server_key_version: "encryptionkey@0"
I20260620 12:24:07.455174 22798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:07.476431 22798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:07.479287 22798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:07.479489 22798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:07.486617 22798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38619
I20260620 12:24:07.486919 22865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38619 every 8 connection(s)
I20260620 12:24:07.487547 22798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:24:07.490232 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:07.490765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 22798
I20260620 12:24:07.490876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781957967886955-392-0/minicluster-data/master-2/wal/instance
I20260620 12:24:07.491204 392 external_mini_cluster.cc:1428] Setting key 6be6d9e85a045f393878318e726a8868
I20260620 12:24:07.494536 22866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:07.505941 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.496404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.0.98.62:33917)
0620 12:24:07.496572 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.496577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.496818 (+ 241us) server_negotiation.cc:407] Connection header received
0620 12:24:07.496949 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.496952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.497014 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.497148 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.498032 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.498586 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.499567 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.499807 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.499931 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.502448 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.502456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.502458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.502483 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.504034 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.504607 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.504610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.504611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.504681 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.504963 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.504965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.504967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.505243 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.505335 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.505734 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.505825 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 12:24:07.506635 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.496324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33917 (local address 127.0.0.1:33842)
0620 12:24:07.496684 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.496728 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.496846 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.497194 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.497200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.497646 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.497798 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.497805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.498733 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.498737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.499427 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.499436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.499964 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.499969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.500091 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.500652 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.500668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.502292 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.504153 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.504158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.504162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.504495 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.504773 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.504777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.504779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.504868 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.505341 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.505347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.505461 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.505681 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.505838 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":121,"threads_started":1}
I20260620 12:24:07.506635 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.492565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.0.98.60:38619)
0620 12:24:07.492999 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.493017 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.493077 (+ 60us) server_negotiation.cc:407] Connection header received
0620 12:24:07.493920 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.493942 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.494332 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.494694 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.495549 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.496848 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.498097 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.498565 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.498729 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.501321 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.501327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.501345 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.501365 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.503221 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.503857 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.503861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.503865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.503987 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.504309 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.504312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.504314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.504849 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.505129 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.505352 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.505627 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260620 12:24:07.507870 22866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:07.515877 22791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } attempt: 3
I20260620 12:24:07.517318 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.508395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37099 (local address 127.0.0.1:46674)
0620 12:24:07.508506 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.508521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.508593 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.508988 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.508991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.509226 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.509345 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.509352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.510099 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.510101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.511092 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.511101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.511554 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.511557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.511650 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.512077 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.512091 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.513563 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.515401 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.515406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.515408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.515607 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.515891 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.515894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.515896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.515999 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.516465 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.516468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.516569 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.516992 (+ 423us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.517159 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 12:24:07.517318 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.508595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.0.98.61:37099)
0620 12:24:07.508737 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.508742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.508755 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:24:07.508806 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.508809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.508848 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.508959 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.509444 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.509975 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.511214 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.511431 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.511548 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.513718 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.513722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.513723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.513764 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.515302 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.515709 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.515713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.515715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.515792 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.516108 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.516112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.516115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.516347 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.516475 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.517072 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.517181 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 12:24:07.518133 22866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:07.520197 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.509200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.0.98.62:33917)
0620 12:24:07.509330 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.509334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.509345 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:24:07.510103 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.510106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.510145 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.510207 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.511132 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.511649 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.512744 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.513141 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.513253 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.515776 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.515779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.515781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.515801 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.518256 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.518729 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.518735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.518740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.518827 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.519128 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.519132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.519135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.519353 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.519509 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.519907 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.520014 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:24:07.526665 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.516608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:55094)
0620 12:24:07.516709 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.516723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.516791 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.517278 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.517281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.517462 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.517594 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.517602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.519012 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.519015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.519712 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.519720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.520412 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.520416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.520501 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.520950 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.520969 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.522417 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.524983 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.524988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.524990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.525265 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.525592 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.525595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.525597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.525683 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.526167 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.526170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.526270 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.526359 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.526478 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:24:07.527194 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.516883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.0.98.60:38619)
0620 12:24:07.517022 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.517026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.517039 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:24:07.517095 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.517100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.517148 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.517707 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.517780 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.518909 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.519819 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.520133 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.520298 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.522546 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.522550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.522552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.522580 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.524867 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.525387 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.525393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.525395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.525477 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.525819 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.525822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.525823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.526033 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.526833 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.526905 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.527012 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:24:07.527974 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.518555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:55110)
0620 12:24:07.518688 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.518700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.518759 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.519330 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.519333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.519486 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.519570 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.519575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.521389 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.521392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.522243 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.522253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.522849 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.522852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.522928 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.523304 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.523335 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.524591 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.526575 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.526578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.526579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.526811 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.527067 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.527070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.527072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.527141 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.527561 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.527563 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.527649 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.527743 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.527864 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"spinlock_wait_cycles":1920}
I20260620 12:24:07.528393 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.518637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.0.98.60:38619)
0620 12:24:07.519092 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.519103 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.519126 (+ 23us) server_negotiation.cc:407] Connection header received
0620 12:24:07.519158 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.519160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.519203 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.519285 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.519690 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.521279 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.522365 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.522735 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.523288 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.524733 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.524737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.524739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.524760 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.526479 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.526903 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.526906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.526908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.526978 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.527224 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.527230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.527232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.527447 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.527530 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.528157 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.528253 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"spinlock_wait_cycles":5888,"thread_start_us":95,"threads_started":1}
I20260620 12:24:07.530658 22791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff: Bootstrap starting.
I20260620 12:24:07.530992 22866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e: Bootstrap starting.
I20260620 12:24:07.531337 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.521530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.0.98.61:37099)
0620 12:24:07.521625 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.521629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.521773 (+ 144us) server_negotiation.cc:407] Connection header received
0620 12:24:07.521895 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.521898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.521933 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.522021 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.522652 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.523391 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.524279 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.524611 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.524709 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.527478 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.527483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.527491 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.527511 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.529576 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.530038 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.530041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.530043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.530119 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.530408 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.530411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.530413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.530597 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.530848 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.530949 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.531045 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:24:07.533092 22716 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } attempt: 3
I20260620 12:24:07.533481 22791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:07.533479 22866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:07.534373 22791 log.cc:824] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:07.534674 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:24:07.534675 22866 log.cc:824] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:07.536433 22791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff: No bootstrap required, opened a new log
I20260620 12:24:07.536550 22866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e: No bootstrap required, opened a new log
I20260620 12:24:07.537213 22689 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e0f83c1d5d74463b89e9ad1a210e9220"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33844
Jun 20 12:24:07 dist-test-slave-9b3m krb5kdc[22635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:24:07.540537 22866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } }
I20260620 12:24:07.540623 22791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } }
I20260620 12:24:07.540774 22866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880892c0f07e4b2788d1a6985803a17e, State: Initialized, Role: FOLLOWER
I20260620 12:24:07.540841 22791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f53592f58164002b8892bda7d61eeff, State: Initialized, Role: FOLLOWER
I20260620 12:24:07.541127 22866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } }
I20260620 12:24:07.541244 22791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } }
I20260620 12:24:07.541839 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.542119 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:07.542609 22791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:07.542613 22866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:07.543402 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.533472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38619 (local address 127.0.0.1:55116)
0620 12:24:07.533577 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:07.533593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:07.533674 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:07.534691 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:07.534696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:07.534896 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:07.535012 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.535022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.536043 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.536046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.536870 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:07.536881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.537495 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.537499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:07.537584 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.538081 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:07.538096 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:07.539520 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:07.541734 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.541737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.541739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.541975 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.542307 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:07.542310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:07.542312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:07.542398 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:07.542835 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:07.542838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:07.542961 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 12:24:07.543070 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 12:24:07.543205 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 12:24:07.543460 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.533831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.0.98.60:38619)
0620 12:24:07.533941 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.533948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.533960 (+ 12us) server_negotiation.cc:407] Connection header received
0620 12:24:07.534113 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.534116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.534159 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.535130 (+ 971us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.535235 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.535950 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.537001 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.537363 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.539665 (+ 2302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.539771 (+ 106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.539774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.539777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.539794 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.541602 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.542100 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.542103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.542105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.542200 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.542508 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.542511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.542512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.542732 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.542824 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.543136 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.543246 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 12:24:07.543000 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.543725 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:07.546695 22716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220: Bootstrap starting.
I20260620 12:24:07.549432 22716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220: Neither blocks nor log segments found. Creating new log.
W20260620 12:24:07.548182 22897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:24:07.549934 22897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:24:07.550379 22716 log.cc:824] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:07.546682 22723 master_runner.cc:186] Error getting master registration for 127.0.98.62:33917: OK, instance_id { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" instance_seqno: 1781958247066341 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:24:07.550631 22723 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 12:24:07.551599 22891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:24:07.551676 22891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:24:07.552560 22716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220: No bootstrap required, opened a new log
I20260620 12:24:07.551196 22798 master_runner.cc:186] Error getting master registration for 127.0.98.62:33917: OK, instance_id { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" instance_seqno: 1781958247066341 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:24:07.554425 22798 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:24:07.555547 22716 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } }
I20260620 12:24:07.555753 22716 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f83c1d5d74463b89e9ad1a210e9220, State: Initialized, Role: FOLLOWER
I20260620 12:24:07.556067 22716 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } }
I20260620 12:24:07.556484 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.556630 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:07.557247 22716 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:07.557338 22689 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 12:24:07.557466 22689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:07.558689 22689 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } }
I20260620 12:24:07.559617 22689 leader_election.cc:288] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [CANDIDATE]: Term 1 election: Requested vote from peers 2f53592f58164002b8892bda7d61eeff (127.0.98.61:37099), 880892c0f07e4b2788d1a6985803a17e (127.0.98.60:38619)
I20260620 12:24:07.560199 22765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2f53592f58164002b8892bda7d61eeff"
I20260620 12:24:07.560439 22765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:07.561776 22765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f83c1d5d74463b89e9ad1a210e9220 in term 1.
I20260620 12:24:07.562345 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "880892c0f07e4b2788d1a6985803a17e"
I20260620 12:24:07.562539 22840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [term 0 FOLLOWER]: Advancing to term 1
W20260620 12:24:07.562758 22669 tablet.cc:2369] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:24:07.563833 22840 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f83c1d5d74463b89e9ad1a210e9220 in term 1.
I20260620 12:24:07.565022 22664 leader_election.cc:302] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 880892c0f07e4b2788d1a6985803a17e, e0f83c1d5d74463b89e9ad1a210e9220; no voters:
I20260620 12:24:07.565304 22901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:07.565469 22901 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [term 1 LEADER]: Becoming Leader. State: Replica: e0f83c1d5d74463b89e9ad1a210e9220, State: Running, Role: LEADER
I20260620 12:24:07.565748 22901 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } }
W20260620 12:24:07.566260 22912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:24:07.566315 22912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:24:07.567341 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f83c1d5d74463b89e9ad1a210e9220. Latest consensus state: current_term: 1 leader_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.567441 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:07.571556 22648 master_runner.cc:263] No leader master found from master e0f83c1d5d74463b89e9ad1a210e9220
I20260620 12:24:07.571621 22648 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:24:07.571902 22922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:07.572842 22922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:07.578061 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.562906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.0.98.60:38619)
0620 12:24:07.563256 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.563262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.567239 (+ 3977us) server_negotiation.cc:407] Connection header received
0620 12:24:07.567433 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.567436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.567487 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.567548 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.569420 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.570138 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.571237 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.571548 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.571638 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.574263 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.574268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.574270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.574294 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.576150 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.576659 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.576663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.576665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.576749 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.577080 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.577084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.577086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.577299 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.577424 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.577779 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.577897 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260620 12:24:07.582890 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.566632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.0.98.61:37099)
0620 12:24:07.566745 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.566749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.568793 (+ 2044us) server_negotiation.cc:407] Connection header received
0620 12:24:07.568898 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.568901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.568943 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.569027 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.574265 (+ 5238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.575023 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.576210 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.576542 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.576608 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.578243 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.578247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.578250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.578274 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.579919 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.581604 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.581608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.581610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.581709 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.581990 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.581994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.581995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.582191 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.582299 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.582616 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.582741 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 12:24:07.587177 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:07.569175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.98.62:33917)
0620 12:24:07.569291 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.569296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.578887 (+ 9591us) server_negotiation.cc:407] Connection header received
0620 12:24:07.579053 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.579057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.579103 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.579159 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.579965 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.580660 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.581743 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:07.582072 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:07.582144 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:07.583716 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:07.583719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:07.583722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:07.583746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:07.585362 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.585845 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.585849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.585851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.585935 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:07.586227 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:07.586230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:07.586232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:07.586434 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:07.586530 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:07.586905 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 12:24:07.587019 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:24:07.594621 22840 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [term 1 FOLLOWER]: Refusing update from remote peer e0f83c1d5d74463b89e9ad1a210e9220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:07.595306 22919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:07.599555 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f83c1d5d74463b89e9ad1a210e9220. Latest consensus state: current_term: 1 leader_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.599689 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:07.600868 22765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [term 1 FOLLOWER]: Refusing update from remote peer e0f83c1d5d74463b89e9ad1a210e9220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:07.601431 22919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:07.604826 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f83c1d5d74463b89e9ad1a210e9220. Latest consensus state: current_term: 1 leader_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.604949 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:07.609439 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.609565 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:07.611048 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.611164 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:07.611526 22924 mvcc.cc:205] Tried to move back new op lower bound from 7298900982107504640 to 7298900982032359424. Current Snapshot: MvccSnapshot[applied={T|T < 7298900982107504640}]
I20260620 12:24:07.611661 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.611768 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220 [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:07.612674 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f83c1d5d74463b89e9ad1a210e9220" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } peers { permanent_uuid: "2f53592f58164002b8892bda7d61eeff" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37099 } } peers { permanent_uuid: "880892c0f07e4b2788d1a6985803a17e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38619 } } }
I20260620 12:24:07.612780 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:07.613422 22925 mvcc.cc:205] Tried to move back new op lower bound from 7298900982107504640 to 7298900982032359424. Current Snapshot: MvccSnapshot[applied={T|T < 7298900982107504640}]
I20260620 12:24:07.613586 22927 mvcc.cc:205] Tried to move back new op lower bound from 7298900982107504640 to 7298900982032359424. Current Snapshot: MvccSnapshot[applied={T|T < 7298900982107504640}]
I20260620 12:24:07.621505 22922 catalog_manager.cc:1347] Generated new cluster ID: 905f727ae45f4c6c9c971b4e106b3927
I20260620 12:24:07.621578 22922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:07.641479 22922 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:07.643047 22922 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:07.671767 22922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0f83c1d5d74463b89e9ad1a210e9220: Generated new TSK 0
I20260620 12:24:07.672629 22922 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:24:07.704211 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.701011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.0.98.61:37099)
0620 12:24:07.701174 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.701179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.703046 (+ 1867us) server_negotiation.cc:407] Connection header received
0620 12:24:07.703251 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.703254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.703303 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.703365 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.704074 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46698: BlockingRecv error: recv got EOF from 127.0.0.1:46698 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 12:24:07.705631 22720 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.701815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.0.98.62:33917)
0620 12:24:07.701937 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.701942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.704732 (+ 2790us) server_negotiation.cc:407] Connection header received
0620 12:24:07.704912 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.704915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.704963 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.705014 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.705517 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33852: BlockingRecv error: recv got EOF from 127.0.0.1:33852 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 12:24:07.707031 22871 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.702506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.0.98.60:38619)
0620 12:24:07.702648 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.702653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.706098 (+ 3445us) server_negotiation.cc:407] Connection header received
0620 12:24:07.706340 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.706343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.706393 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.706443 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.706920 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55126: BlockingRecv error: recv got EOF from 127.0.0.1:55126 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 12:24:07.711506 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.709330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.0.98.61:37099)
0620 12:24:07.709430 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.709435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.710536 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 12:24:07.710793 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.710796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.710872 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.710931 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.711413 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46700: BlockingRecv error: recv got EOF from 127.0.0.1:46700 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 12:24:07.716208 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.714754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.0.98.61:37099)
0620 12:24:07.714845 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.714849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.715354 (+ 505us) server_negotiation.cc:407] Connection header received
0620 12:24:07.715537 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.715540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.715586 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.715639 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.716122 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46708: BlockingRecv error: recv got EOF from 127.0.0.1:46708 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 12:24:07.721495 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.720009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.0.98.61:37099)
0620 12:24:07.720102 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.720106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.720618 (+ 512us) server_negotiation.cc:407] Connection header received
0620 12:24:07.720819 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.720822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.720869 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.720921 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.721409 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46724: BlockingRecv error: recv got EOF from 127.0.0.1:46724 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 12:24:07.728920 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.727349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.0.98.61:37099)
0620 12:24:07.727484 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.727489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.728021 (+ 532us) server_negotiation.cc:407] Connection header received
0620 12:24:07.728211 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.728215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.728265 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.728319 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.728824 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46734: BlockingRecv error: recv got EOF from 127.0.0.1:46734 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 12:24:07.740593 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.739049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.0.98.61:37099)
0620 12:24:07.739156 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.739161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.739686 (+ 525us) server_negotiation.cc:407] Connection header received
0620 12:24:07.739872 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.739876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.739924 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.739972 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.740487 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46742: BlockingRecv error: recv got EOF from 127.0.0.1:46742 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 12:24:07.760851 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.759251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.0.98.61:37099)
0620 12:24:07.759393 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.759400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.759735 (+ 335us) server_negotiation.cc:407] Connection header received
0620 12:24:07.759980 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.759985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.760060 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.760168 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.760718 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46746: BlockingRecv error: recv got EOF from 127.0.0.1:46746 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 12:24:07.795867 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.794824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.0.98.61:37099)
0620 12:24:07.794974 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.794978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.795116 (+ 138us) server_negotiation.cc:407] Connection header received
0620 12:24:07.795231 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.795234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.795281 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.795373 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.795771 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46750: BlockingRecv error: recv got EOF from 127.0.0.1:46750 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 12:24:07.863128 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.862038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.0.98.61:37099)
0620 12:24:07.862180 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.862184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.862339 (+ 155us) server_negotiation.cc:407] Connection header received
0620 12:24:07.862461 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.862464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.862512 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.862598 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.862975 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46760: BlockingRecv error: recv got EOF from 127.0.0.1:46760 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 12:24:07.994701 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:07.993436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46764 (local address 127.0.98.61:37099)
0620 12:24:07.993605 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:07.993610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:07.993752 (+ 142us) server_negotiation.cc:407] Connection header received
0620 12:24:07.993928 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:07.993931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:07.994011 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:07.994103 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:07.994568 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46764: BlockingRecv error: recv got EOF from 127.0.0.1:46764 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 12:24:08.254045 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:08.252915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.0.98.61:37099)
0620 12:24:08.253134 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:08.253138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:08.253154 (+ 16us) server_negotiation.cc:407] Connection header received
0620 12:24:08.253246 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:08.253249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:08.253296 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:08.253371 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:08.253894 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46768: BlockingRecv error: recv got EOF from 127.0.0.1:46768 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 12:24:08.551218 22897 catalog_manager.cc:1259] Loaded cluster ID: 905f727ae45f4c6c9c971b4e106b3927
I20260620 12:24:08.551354 22897 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e: loading cluster ID for follower catalog manager: success
I20260620 12:24:08.552887 22891 catalog_manager.cc:1259] Loaded cluster ID: 905f727ae45f4c6c9c971b4e106b3927
I20260620 12:24:08.552948 22891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff: loading cluster ID for follower catalog manager: success
I20260620 12:24:08.553737 22897 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e: acquiring CA information for follower catalog manager: success
I20260620 12:24:08.554917 22891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff: acquiring CA information for follower catalog manager: success
I20260620 12:24:08.555092 22897 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 880892c0f07e4b2788d1a6985803a17e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:24:08.555955 22891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f53592f58164002b8892bda7d61eeff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:24:08.771963 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:08.768104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.0.98.61:37099)
0620 12:24:08.768338 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:08.768342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:08.768359 (+ 17us) server_negotiation.cc:407] Connection header received
0620 12:24:08.768517 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:08.768520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:08.768655 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:08.768812 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:08.769030 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.769614 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.770461 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.770703 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.770818 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:08.771102 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:08.771539 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:08.771667 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:08.771775 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 12:24:08.771828 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260620 12:24:08.778622 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:08.775048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.0.98.60:38619)
0620 12:24:08.775279 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:08.775282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:08.775296 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:24:08.775466 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:08.775469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:08.775581 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:08.775675 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:08.775893 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.776385 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.777285 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.777498 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.777572 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:08.777852 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:08.778245 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:08.778334 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:08.778458 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 12:24:08.778505 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260620 12:24:08.786000 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:08.781589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.0.98.61:37099)
0620 12:24:08.781711 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:08.781718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:08.782315 (+ 597us) server_negotiation.cc:407] Connection header received
0620 12:24:08.782445 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:08.782451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:08.782505 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:08.782574 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:08.782955 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.783458 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.784728 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.785097 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.785210 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:08.785501 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:08.785587 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:08.785652 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:08.785802 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 12:24:08.785866 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:24:08.786072 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:08.781962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.0.98.60:38619)
0620 12:24:08.782075 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:08.782080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:08.782159 (+ 79us) server_negotiation.cc:407] Connection header received
0620 12:24:08.782370 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:08.782376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:08.782435 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:08.782487 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:08.782930 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.783679 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.784846 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.785128 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.785250 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:08.785651 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:08.785723 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:08.785796 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:08.785911 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 12:24:08.785956 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 12:24:08.787052 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:08.781590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.0.98.62:33917)
0620 12:24:08.781820 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:08.781825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:08.782036 (+ 211us) server_negotiation.cc:407] Connection header received
0620 12:24:08.782229 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:08.782232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:08.782393 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:08.782504 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:08.782900 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.783706 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.784781 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:08.785139 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:08.785224 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:08.785635 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:08.785976 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:08.786216 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:08.786822 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 12:24:08.786883 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260620 12:24:08.793375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 22648
I20260620 12:24:08.799928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 22723
I20260620 12:24:08.805699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 22798
2026-06-20T12:24:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3061 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 12:24:08 dist-test-slave-9b3m krb5kdc[23014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 12:24:08 dist-test-slave-9b3m krb5kdc[23014](info): set up 2 sockets
Jun 20 12:24:08 dist-test-slave-9b3m krb5kdc[23014](info): commencing operation
krb5kdc: starting...
W20260620 12:24:09.849515 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/test-user.keytab.
Jun 20 12:24:09 dist-test-slave-9b3m krb5kdc[23014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T12:24:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T12:24:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 12:24:09.967849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45693
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45693,127.0.98.61:36335,127.0.98.60:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 12:24:10.083498 23027 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:24:10.083756 23027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:10.083778 23027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:10.083822 23027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:10.088375 23027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:10.088429 23027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:10.088449 23027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:10.088464 23027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:10.088479 23027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:10.093456 23027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45693,127.0.98.61:36335,127.0.98.60:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:10.094350 23027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:10.095263 23027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:10.103260 23033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:10.103260 23032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:10.103286 23035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:10.103849 23027 server_base.cc:1034] running on GCE node
I20260620 12:24:10.104377 23027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:10.105504 23027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:10.106714 23027 hybrid_clock.cc:648] HybridClock initialized: now 1781958250106680 us; error 46 us; skew 500 ppm
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:10.109644 23027 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 12:24:10.110538 23027 webserver.cc:460] Webserver started at http://127.0.98.62:44571/ using document root <none> and password file <none>
I20260620 12:24:10.111094 23027 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:10.111188 23027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:10.111413 23027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:10.113428 23027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/data/instance:
uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "914275445459d7e55bedd5ed787f4d67"
server_key_iv: "f2f99c2fa311cb0cb6b761720c6e6fce"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.113969 23027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/wal/instance:
uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "914275445459d7e55bedd5ed787f4d67"
server_key_iv: "f2f99c2fa311cb0cb6b761720c6e6fce"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.117491 23027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 12:24:10.119756 23042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:10.120971 23027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:24:10.121160 23027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/wal
uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "914275445459d7e55bedd5ed787f4d67"
server_key_iv: "f2f99c2fa311cb0cb6b761720c6e6fce"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.121272 23027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:10.132800 23027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:10.135969 23027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:10.136170 23027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:10.143378 23027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45693
I20260620 12:24:10.143730 23094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45693 every 8 connection(s)
I20260620 12:24:10.144544 23027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/data/info.pb
I20260620 12:24:10.147435 23095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:10.151048 23095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:10.153849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 23027
I20260620 12:24:10.154027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-0/wal/instance
I20260620 12:24:10.154315 392 external_mini_cluster.cc:1428] Setting key bb685f6e7e73fdcf71c7ffc75255674d
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 12:24:10.167924 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.153667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.98.62:45693)
0620 12:24:10.154620 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.154629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.154665 (+ 36us) server_negotiation.cc:407] Connection header received
0620 12:24:10.156101 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.156111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.156403 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.156609 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.157490 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.158365 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.160012 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.160368 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.160525 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.163256 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.163261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.163270 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.163290 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.164814 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.165360 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.165364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.165365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.165471 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.165793 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.165796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.165797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.166148 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.166288 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.166666 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.166905 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":855,"spinlock_wait_cycles":73472,"thread_start_us":140,"threads_started":1}
I20260620 12:24:10.167924 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.153667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45693 (local address 127.0.0.1:58292)
0620 12:24:10.154289 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.154344 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.155650 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.156616 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.156620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.156947 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.157287 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.157302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.158528 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.158534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.159853 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.159867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.160554 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.160558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.160730 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.161384 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.161410 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.163106 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.164923 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.164933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.164943 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.165243 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.165577 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.165581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.165583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.165694 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.166260 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.166267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.166405 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.166567 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.166955 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":172,"thread_start_us":134,"threads_started":1}
I20260620 12:24:10.170302 23095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:10.171064 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.170789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38666)
0620 12:24:10.170902 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.170974 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 12:24:10.171305 23045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
I20260620 12:24:10.172125 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.171906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38668)
0620 12:24:10.172035 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.172076 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 12:24:10.172355 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36335: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 12:24:10.180409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36335
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45693,127.0.98.61:36335,127.0.98.60:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:24:10.221621 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } attempt: 1
I20260620 12:24:10.222538 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.222225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38674)
0620 12:24:10.222382 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.222459 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:24:10.223328 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.223120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38688)
0620 12:24:10.223229 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.223279 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 12:24:10.223455 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36335: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
W20260620 12:24:10.288110 23102 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:24:10.288401 23102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:10.288466 23102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:10.288542 23102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 12:24:10.291612 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } attempt: 2
I20260620 12:24:10.292388 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.292101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38700)
0620 12:24:10.292249 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.292316 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 12:24:10.292667 23102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:10.292725 23102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:10.292752 23102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:10.292779 23102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:10.292815 23102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:10.293197 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.292933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38710)
0620 12:24:10.293089 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.293145 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 12:24:10.293322 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36335: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36335: connect: Connection refused (error 111)
I20260620 12:24:10.298130 23102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45693,127.0.98.61:36335,127.0.98.60:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:10.299081 23102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 12:24:10.300175 23102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 12:24:10.308622 23107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:10.308717 23102 server_base.cc:1034] running on GCE node
W20260620 12:24:10.308627 23110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:10.308633 23108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:10.309437 23102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:10.310549 23102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:10.311743 23102 hybrid_clock.cc:648] HybridClock initialized: now 1781958250311725 us; error 28 us; skew 500 ppm
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:10.314409 23102 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 12:24:10.315251 23102 webserver.cc:460] Webserver started at http://127.0.98.61:34571/ using document root <none> and password file <none>
I20260620 12:24:10.315795 23102 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:10.315886 23102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:10.316114 23102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:10.318150 23102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/data/instance:
uuid: "d88af46f8fcf47dba89c883089b03b8d"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "cb1c043d36625a7af8bfe8bb65b6f6f3"
server_key_iv: "3c89bdafa5025d3d10061b777d46e0ac"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.318696 23102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/wal/instance:
uuid: "d88af46f8fcf47dba89c883089b03b8d"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "cb1c043d36625a7af8bfe8bb65b6f6f3"
server_key_iv: "3c89bdafa5025d3d10061b777d46e0ac"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.322214 23102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 12:24:10.324542 23117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:10.325845 23102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:24:10.325973 23102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/wal
uuid: "d88af46f8fcf47dba89c883089b03b8d"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "cb1c043d36625a7af8bfe8bb65b6f6f3"
server_key_iv: "3c89bdafa5025d3d10061b777d46e0ac"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.326087 23102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:10.344074 23102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:10.347011 23102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:10.347210 23102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:10.353777 23102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36335
I20260620 12:24:10.354054 23169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36335 every 8 connection(s)
I20260620 12:24:10.354718 23102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/data/info.pb
I20260620 12:24:10.356184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 23102
I20260620 12:24:10.356307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-1/wal/instance
I20260620 12:24:10.356637 392 external_mini_cluster.cc:1428] Setting key e1362e171c487050d295c2914f9cdcd9
I20260620 12:24:10.357885 23170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:10.362013 23170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:10.376816 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.364280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.0.98.62:45693)
0620 12:24:10.364465 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.364470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.365260 (+ 790us) server_negotiation.cc:407] Connection header received
0620 12:24:10.366256 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.366259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.366304 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.366406 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.367763 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.368325 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.369665 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.370119 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.370246 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.372945 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.372949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.372952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.373038 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.374787 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.375267 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.375273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.375275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.375375 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.375687 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.375689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.375690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.375877 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.375956 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.376529 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.376630 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 12:24:10.377641 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.364607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45693 (local address 127.0.0.1:58300)
0620 12:24:10.365107 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.365159 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.366061 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.366694 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.366701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.367029 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.367567 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.367602 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.368464 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.368471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.369540 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.369550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.370275 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.370278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.370410 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.371060 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.371076 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.372776 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.374903 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.374907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.374918 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.375161 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.375468 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.375470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.375472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.375599 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.375974 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.375979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.376268 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.376430 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.376722 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":145,"threads_started":1}
I20260620 12:24:10.379145 23170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 12:24:10.383601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZkvxte/build/debug/bin/kudu
/tmp/dist-test-taskZkvxte/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42123
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45693,127.0.98.61:36335,127.0.98.60:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 12:24:10.384471 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } attempt: 3
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:24:10.388980 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.379624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38720)
0620 12:24:10.379732 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.379747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.379876 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.380490 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.380492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.380687 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.380770 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.380776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.381922 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.381925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.382474 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.382482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.383204 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.383207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.383316 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.383844 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.383861 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.385318 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.387462 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.387465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.387467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.387689 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.387951 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.387953 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.387955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.388020 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.388476 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.388479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.388555 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.388671 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.388800 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 12:24:10.389564 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.379784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.0.98.61:36335)
0620 12:24:10.380161 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.380170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.380188 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:24:10.380219 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.380223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.380358 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.380471 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.380871 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.381826 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.382641 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.383092 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.383207 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.385428 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.385438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.385454 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.385480 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.387351 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.387793 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.387798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.387800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.387869 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.388117 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.388120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.388121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.388368 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.388512 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.388762 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.388894 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":258,"thread_start_us":161,"threads_started":1}
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:24:10.391619 23170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:10.392333 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.392102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53302)
0620 12:24:10.392212 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.392277 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 12:24:10.392550 23119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
I20260620 12:24:10.393483 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.393225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53306)
0620 12:24:10.393361 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.393415 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 12:24:10.393666 23170 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
I20260620 12:24:10.395217 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.384894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38722)
0620 12:24:10.385083 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.385098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.385176 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.385656 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.385661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.385830 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.385944 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.385954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.387097 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.387104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.388010 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.388020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.388753 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.388759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.388899 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.389372 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.389392 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.390853 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.393808 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.393812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.393817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.394010 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.394284 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.394289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.394291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.394372 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.394731 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.394734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.394827 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.394935 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.395065 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 12:24:10.395231 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.385169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.0.98.61:36335)
0620 12:24:10.385447 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.385459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.385477 (+ 18us) server_negotiation.cc:407] Connection header received
0620 12:24:10.385503 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.385505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.385537 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.385641 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.386074 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.386956 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.388129 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.388595 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.389301 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.390996 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.390999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.391002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.391024 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.393696 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.394106 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.394111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.394112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.394194 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.394474 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.394478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.394479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.394638 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.394739 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.394982 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.395084 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":106,"threads_started":1}
I20260620 12:24:10.396005 23095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:10.396668 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.396439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53316)
0620 12:24:10.396547 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.396613 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 12:24:10.397392 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.397156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53318)
0620 12:24:10.397286 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.397332 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 12:24:10.397567 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
I20260620 12:24:10.428718 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } attempt: 1
I20260620 12:24:10.429589 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.429245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53320)
0620 12:24:10.429433 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.429511 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 12:24:10.430305 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.430099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53326)
0620 12:24:10.430207 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.430256 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 12:24:10.430428 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
I20260620 12:24:10.442910 23170 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } attempt: 1
I20260620 12:24:10.443779 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.443482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53342)
0620 12:24:10.443646 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.443707 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 12:24:10.444561 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.444372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53352)
0620 12:24:10.444466 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.444511 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 12:24:10.444743 23170 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
W20260620 12:24:10.496119 23176 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 12:24:10.496376 23176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 12:24:10.496398 23176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 12:24:10.496491 23176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 12:24:10.500560 23176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 12:24:10.500615 23176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 12:24:10.500635 23176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 12:24:10.500651 23176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 12:24:10.500712 23176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 12:24:10.505622 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } attempt: 2
I20260620 12:24:10.506157 23176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45693,127.0.98.61:36335,127.0.98.60:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 11:43:30 UTC on ccycloud.quasar-ptebcd.root.comops.site
build id 4457460
I20260620 12:24:10.506431 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.506143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53360)
0620 12:24:10.506281 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.506349 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 12:24:10.507195 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.506988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53368)
0620 12:24:10.507092 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.507139 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 12:24:10.507301 23176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260620 12:24:10.507388 23095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
I20260620 12:24:10.508323 23176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 12:24:10.512915 23170 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } attempt: 2
I20260620 12:24:10.513595 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.513358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53376)
0620 12:24:10.513477 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.513532 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 12:24:10.514273 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.514093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53390)
0620 12:24:10.514187 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.514227 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 12:24:10.514446 23170 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42123: connect: Connection refused (error 111)
W20260620 12:24:10.517005 23183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:10.517064 23186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 12:24:10.517131 23184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 12:24:10.517412 23176 server_base.cc:1034] running on GCE node
I20260620 12:24:10.517920 23176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 12:24:10.518968 23176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 12:24:10.520121 23176 hybrid_clock.cc:648] HybridClock initialized: now 1781958250520085 us; error 51 us; skew 500 ppm
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 12:24:10.522749 23176 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 12:24:10.523556 23176 webserver.cc:460] Webserver started at http://127.0.98.60:43981/ using document root <none> and password file <none>
I20260620 12:24:10.524111 23176 fs_manager.cc:359] Metadata directory not provided
I20260620 12:24:10.524171 23176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 12:24:10.524325 23176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 12:24:10.526782 23176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/data/instance:
uuid: "55b4bd5165a641d5a554650a8e6eb958"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "8de291c854038400f0552aaa21009c14"
server_key_iv: "c0185cf4865bf215f07221b3c8981961"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.527258 23176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/wal/instance:
uuid: "55b4bd5165a641d5a554650a8e6eb958"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "8de291c854038400f0552aaa21009c14"
server_key_iv: "c0185cf4865bf215f07221b3c8981961"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.530807 23176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 12:24:10.532989 23193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 12:24:10.534159 23176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 12:24:10.534278 23176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/wal
uuid: "55b4bd5165a641d5a554650a8e6eb958"
format_stamp: "Formatted at 2026-06-20 12:24:10 on dist-test-slave-9b3m"
server_key: "8de291c854038400f0552aaa21009c14"
server_key_iv: "c0185cf4865bf215f07221b3c8981961"
server_key_version: "encryptionkey@0"
I20260620 12:24:10.534373 23176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 12:24:10.545723 23176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 12:24:10.548839 23176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 12:24:10.549080 23176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 12:24:10.555675 23176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42123
I20260620 12:24:10.555946 23245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42123 every 8 connection(s)
I20260620 12:24:10.556659 23176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/data/info.pb
I20260620 12:24:10.559458 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 12:24:10.560048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZkvxte/build/debug/bin/kudu as pid 23176
I20260620 12:24:10.560192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZkvxte/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781957967886955-392-0/minicluster-data/master-2/wal/instance
I20260620 12:24:10.560431 392 external_mini_cluster.cc:1428] Setting key a7c8bbe27e29ae2ada7f00800b2ab63e
I20260620 12:24:10.563975 23246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:10.575194 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.565660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.0.98.62:45693)
0620 12:24:10.565796 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.565801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.566068 (+ 267us) server_negotiation.cc:407] Connection header received
0620 12:24:10.566191 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.566194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.566249 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.566332 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.566941 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.567450 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.568385 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.568771 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.568890 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.571264 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.571268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.571270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.571299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.573420 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.573882 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.573886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.573887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.573974 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.574284 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.574287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.574288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.574525 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.574622 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.574938 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.575055 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 12:24:10.575382 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.565572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45693 (local address 127.0.0.1:58302)
0620 12:24:10.565934 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.565981 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.566098 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.566388 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.566392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.566704 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.566817 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.566822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.567579 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.567582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.568270 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.568278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.568919 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.568921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.568995 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.569561 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.569580 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.571125 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.573520 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.573529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.573535 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.573751 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.574083 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.574086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.574088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.574185 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.574629 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.574634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.574727 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.574835 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.574957 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":4,"thread_start_us":145,"threads_started":1}
I20260620 12:24:10.575392 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.562076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.0.98.60:42123)
0620 12:24:10.562579 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.562599 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.562642 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:24:10.563359 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.563375 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.563702 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.564025 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.564810 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.565997 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.567012 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.567471 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.567661 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.570607 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.570614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.570627 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.570652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.572389 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.572870 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.572874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.572879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.573019 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.573351 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.573354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.573355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.573707 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.573893 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.574204 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.574484 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":105,"threads_started":1}
I20260620 12:24:10.576563 23246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 12:24:10.585263 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.577071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36335 (local address 127.0.0.1:38728)
0620 12:24:10.577295 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.577309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.577389 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.577627 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.577630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.577793 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.577908 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.577915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.578788 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.578790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.579322 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.579331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.579798 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.579802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.579904 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.580306 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.580320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.581825 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.583662 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.583665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.583666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.583937 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.584244 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.584246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.584247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.584356 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.584819 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.584822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.584896 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.584999 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.585151 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 12:24:10.585421 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.577183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.0.98.61:36335)
0620 12:24:10.577290 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.577294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.577352 (+ 58us) server_negotiation.cc:407] Connection header received
0620 12:24:10.577477 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.577481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.577532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.577617 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.578049 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.578675 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.579434 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.579673 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.579751 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.581959 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.581962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.581965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.581996 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.583548 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.584046 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.584051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.584056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.584147 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.584459 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.584462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.584464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.584696 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.584803 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.585120 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.585239 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 12:24:10.586138 23246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
I20260620 12:24:10.587355 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.577735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.0.98.62:45693)
0620 12:24:10.577839 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.577843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.577886 (+ 43us) server_negotiation.cc:407] Connection header received
0620 12:24:10.578032 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.578036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.578078 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.578163 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.578668 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.579187 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.580431 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.580648 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.580725 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.583454 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.583459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.583462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.583489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.585243 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.585905 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.585908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.585910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.585995 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.586374 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.586378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.586380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.586583 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.586825 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.586995 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.587109 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 12:24:10.595664 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.586644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53408)
0620 12:24:10.586816 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.586830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.586900 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.587129 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.587132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.587279 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.587334 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.587339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.588791 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.588796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.589678 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.589688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.590290 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.590292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.590375 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.590772 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.590786 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.592172 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.594059 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.594062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.594064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.594288 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.594579 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.594581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.594583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.594652 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.595090 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.595093 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.595195 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.595316 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.595459 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":16}
I20260620 12:24:10.595777 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.586707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.0.98.60:42123)
0620 12:24:10.586955 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.586960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.586973 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:24:10.586999 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.587001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.587034 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.587120 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.587638 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.588659 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.589810 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.590188 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.590282 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.592374 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.592379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.592384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.592408 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.593937 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.594383 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.594389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.594392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.594484 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.594751 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.594754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.594756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.594985 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.595084 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.595366 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.595531 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":175}
I20260620 12:24:10.598014 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.588516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.0.98.61:36335)
0620 12:24:10.588615 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.588619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.588717 (+ 98us) server_negotiation.cc:407] Connection header received
0620 12:24:10.588819 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.588821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.588857 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.588913 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.589645 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.590334 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.591432 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.591778 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.591901 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.594663 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.594666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.594668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.594690 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.596445 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.596851 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.596856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.596857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.596938 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.597236 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.597239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.597241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.597408 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.597627 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.597750 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.597856 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 12:24:10.598790 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958: Bootstrap starting.
I20260620 12:24:10.601258 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 12:24:10.601441 23246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:10.602147 23246 log.cc:824] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:10.602389 23069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58314
I20260620 12:24:10.603952 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958: No bootstrap required, opened a new log
I20260620 12:24:10.605598 23170 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } attempt: 3
I20260620 12:24:10.606525 23095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } attempt: 3
I20260620 12:24:10.607641 23246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } }
I20260620 12:24:10.607878 23246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b4bd5165a641d5a554650a8e6eb958, State: Initialized, Role: FOLLOWER
I20260620 12:24:10.608260 23246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } }
I20260620 12:24:10.608774 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.609023 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:10.610008 23246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 12:24:10 dist-test-slave-9b3m krb5kdc[23014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781958250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 12:24:10.614773 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.605982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53424)
0620 12:24:10.606127 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.606145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.606240 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.606472 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.606475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.606655 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.606766 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.606777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.607686 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.607690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.608312 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.608321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.608861 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.608865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.608946 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.609499 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.609514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.610991 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.613144 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.613147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.613149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.613425 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.613692 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.613697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.613700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.613794 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.614273 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.614276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.614366 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.614468 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.614605 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 12:24:10.614800 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.606131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.0.98.60:42123)
0620 12:24:10.606237 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.606241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.606255 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:24:10.606343 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.606346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.606385 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.606463 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.606897 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.607603 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.608429 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.608765 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.608861 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.611141 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.611144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.611147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.611171 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.612994 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.613512 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.613515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.613517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.613602 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.613895 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.613901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.613903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.614140 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.614229 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.614548 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.614661 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":2432}
W20260620 12:24:10.617997 23264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:24:10.618070 23264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:24:10.618151 23170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d: Bootstrap starting.
I20260620 12:24:10.616071 23176 master_runner.cc:186] Error getting master registration for 127.0.98.62:45693: OK, instance_id { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" instance_seqno: 1781958250141400 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:24:10.619426 23176 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:24:10.620100 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.606878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42123 (local address 127.0.0.1:53438)
0620 12:24:10.607034 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 12:24:10.607046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 12:24:10.607118 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 12:24:10.607580 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 12:24:10.607583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 12:24:10.607779 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 12:24:10.607888 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.607899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.609774 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.609778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.610371 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 12:24:10.610379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.610985 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.610988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 12:24:10.611083 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.611536 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 12:24:10.611547 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 12:24:10.612759 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 12:24:10.615247 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.615251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.615254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.615507 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.615793 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 12:24:10.615795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 12:24:10.615797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 12:24:10.615883 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 12:24:10.619578 (+ 3695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 12:24:10.619583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 12:24:10.619684 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 12:24:10.619799 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 12:24:10.619932 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 12:24:10.620187 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.606928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.60:42123)
0620 12:24:10.607374 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.607381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.607392 (+ 11us) server_negotiation.cc:407] Connection header received
0620 12:24:10.607421 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.607424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.607467 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.607544 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.608469 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.609678 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.610486 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.610882 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.610978 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.612902 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.612905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.612929 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.612977 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.615143 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.615614 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.615617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.615619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.615696 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.615976 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.615979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.615981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.616196 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.616296 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.619891 (+ 3595us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.620004 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":388,"spinlock_wait_cycles":17664,"thread_start_us":94,"threads_started":1}
I20260620 12:24:10.620746 23170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:10.621608 23170 log.cc:824] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:10.623143 23095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b: Bootstrap starting.
I20260620 12:24:10.623384 23170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d: No bootstrap required, opened a new log
I20260620 12:24:10.625988 23095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b: Neither blocks nor log segments found. Creating new log.
I20260620 12:24:10.626817 23095 log.cc:824] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b: Log is configured to *not* fsync() on all Append() calls
I20260620 12:24:10.627697 23170 raft_consensus.cc:348] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } }
I20260620 12:24:10.627908 23170 raft_consensus.cc:729] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88af46f8fcf47dba89c883089b03b8d, State: Initialized, Role: FOLLOWER
I20260620 12:24:10.628239 23170 consensus_queue.cc:260] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } }
I20260620 12:24:10.628754 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.628917 23095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b: No bootstrap required, opened a new log
I20260620 12:24:10.629072 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:10.629621 23170 sys_catalog.cc:564] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:10.631947 23095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } }
I20260620 12:24:10.632174 23095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f3d26c748e45c48abf8b4bb2cb1e6b, State: Initialized, Role: FOLLOWER
I20260620 12:24:10.632476 23095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } }
I20260620 12:24:10.633710 23095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 12:24:10.633781 23069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 12:24:10.633891 23069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:10.635089 23069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } }
I20260620 12:24:10.636006 23069 leader_election.cc:288] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [CANDIDATE]: Term 1 election: Requested vote from peers d88af46f8fcf47dba89c883089b03b8d (127.0.98.61:36335), 55b4bd5165a641d5a554650a8e6eb958 (127.0.98.60:42123)
I20260620 12:24:10.636440 23144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d88af46f8fcf47dba89c883089b03b8d"
I20260620 12:24:10.636649 23144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:10.637480 23220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "55b4bd5165a641d5a554650a8e6eb958"
I20260620 12:24:10.633328 23102 master_runner.cc:186] Error getting master registration for 127.0.98.62:45693: OK, instance_id { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" instance_seqno: 1781958250141400 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 12:24:10.637715 23220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [term 0 FOLLOWER]: Advancing to term 1
I20260620 12:24:10.637799 23102 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 12:24:10.638376 23048 tablet.cc:2369] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 12:24:10.639003 23220 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70f3d26c748e45c48abf8b4bb2cb1e6b in term 1.
I20260620 12:24:10.639572 23044 leader_election.cc:302] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55b4bd5165a641d5a554650a8e6eb958, 70f3d26c748e45c48abf8b4bb2cb1e6b; no voters:
I20260620 12:24:10.640745 23144 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70f3d26c748e45c48abf8b4bb2cb1e6b in term 1.
W20260620 12:24:10.644341 23291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:24:10.644413 23291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:24:10.645414 23027 master_runner.cc:263] No leader master found from master 70f3d26c748e45c48abf8b4bb2cb1e6b
I20260620 12:24:10.645478 23027 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 12:24:10.645550 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.645648 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:10.645721 23293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 12:24:10.645787 23293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [term 1 LEADER]: Becoming Leader. State: Replica: 70f3d26c748e45c48abf8b4bb2cb1e6b, State: Running, Role: LEADER
I20260620 12:24:10.646080 23293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } }
I20260620 12:24:10.650359 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f3d26c748e45c48abf8b4bb2cb1e6b. Latest consensus state: current_term: 1 leader_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.650470 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: This master's current role is: LEADER
W20260620 12:24:10.653206 23279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 12:24:10.653277 23279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 12:24:10.657279 23303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 12:24:10.658186 23303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 12:24:10.667116 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.642068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.0.98.62:45693)
0620 12:24:10.642168 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.642173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.644869 (+ 2696us) server_negotiation.cc:407] Connection header received
0620 12:24:10.645787 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.645790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.645836 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.645886 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.650611 (+ 4725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.651397 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.655836 (+ 4439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.656166 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.656241 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.660312 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.660316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.660319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.660345 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.662068 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.665414 (+ 3346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.665418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.665420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.665510 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.665790 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.665793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.665795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.666005 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.666115 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.666836 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.666956 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:24:10.669731 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.642650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.0.98.60:42123)
0620 12:24:10.642757 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.642763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.647650 (+ 4887us) server_negotiation.cc:407] Connection header received
0620 12:24:10.647810 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.647813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.647858 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.647906 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.652607 (+ 4701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.653370 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.657438 (+ 4068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.657816 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.657898 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.662582 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.662586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.662589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.662618 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.664938 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.666399 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.666403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.666405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.666501 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.668684 (+ 2183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.668688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.668690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.668899 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.669025 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.669423 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.669558 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 12:24:10.673909 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:10.641363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.0.98.61:36335)
0620 12:24:10.641479 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.641484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.643251 (+ 1767us) server_negotiation.cc:407] Connection header received
0620 12:24:10.644946 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.644949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.645001 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.645076 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.648831 (+ 3755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.649668 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.654882 (+ 5214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:10.655219 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:10.655295 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:10.670089 (+ 14794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 12:24:10.670094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 12:24:10.670097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 12:24:10.670128 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 12:24:10.671809 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.672264 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.672268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.672270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.672356 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 12:24:10.672647 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 12:24:10.672651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 12:24:10.672652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 12:24:10.672854 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 12:24:10.672961 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:10.673658 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 12:24:10.673772 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 12:24:10.676645 23220 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [term 1 FOLLOWER]: Refusing update from remote peer 70f3d26c748e45c48abf8b4bb2cb1e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:10.676645 23144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [term 1 FOLLOWER]: Refusing update from remote peer 70f3d26c748e45c48abf8b4bb2cb1e6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 12:24:10.677403 23293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:10.677614 23280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 12:24:10.681348 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f3d26c748e45c48abf8b4bb2cb1e6b. Latest consensus state: current_term: 1 leader_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.681479 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:10.682011 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f3d26c748e45c48abf8b4bb2cb1e6b. Latest consensus state: current_term: 1 leader_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.682133 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:10.689174 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.689297 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:10.689410 23304 mvcc.cc:205] Tried to move back new op lower bound from 7298900994764414976 to 7298900994659885056. Current Snapshot: MvccSnapshot[applied={T|T < 7298900994764414976}]
I20260620 12:24:10.689497 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.689600 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:10.691040 23305 mvcc.cc:205] Tried to move back new op lower bound from 7298900994764414976 to 7298900994659885056. Current Snapshot: MvccSnapshot[applied={T|T < 7298900994764414976}]
I20260620 12:24:10.693457 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.693584 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b [sys.catalog]: This master's current role is: LEADER
I20260620 12:24:10.694708 23306 mvcc.cc:205] Tried to move back new op lower bound from 7298900994764414976 to 7298900994659885056. Current Snapshot: MvccSnapshot[applied={T|T < 7298900994764414976}]
I20260620 12:24:10.694811 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70f3d26c748e45c48abf8b4bb2cb1e6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45693 } } peers { permanent_uuid: "d88af46f8fcf47dba89c883089b03b8d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36335 } } peers { permanent_uuid: "55b4bd5165a641d5a554650a8e6eb958" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42123 } } }
I20260620 12:24:10.694928 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 12:24:10.697476 23303 catalog_manager.cc:1347] Generated new cluster ID: d3d4553175e146bf9ecfeaa1060ce982
I20260620 12:24:10.697541 23303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 12:24:10.723861 23303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 12:24:10.725683 23303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 12:24:10.737108 23303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f3d26c748e45c48abf8b4bb2cb1e6b: Generated new TSK 0
I20260620 12:24:10.738026 23303 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 12:24:10.766932 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.763949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.0.98.61:36335)
0620 12:24:10.764074 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.764079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.765886 (+ 1807us) server_negotiation.cc:407] Connection header received
0620 12:24:10.766082 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.766086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.766133 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.766184 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.766800 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38742: BlockingRecv error: recv got EOF from 127.0.0.1:38742 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 12:24:10.768682 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.764717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.0.98.62:45693)
0620 12:24:10.764828 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.764832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.767550 (+ 2718us) server_negotiation.cc:407] Connection header received
0620 12:24:10.767940 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.767943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.767993 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.768043 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.768566 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58330: BlockingRecv error: recv got EOF from 127.0.0.1:58330 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 12:24:10.769868 23252 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.765309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.0.98.60:42123)
0620 12:24:10.765417 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.765421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.767739 (+ 2318us) server_negotiation.cc:407] Connection header received
0620 12:24:10.769163 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.769166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.769214 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.769263 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.769763 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53446: BlockingRecv error: recv got EOF from 127.0.0.1:53446 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 12:24:10.774339 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.772912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.0.98.61:36335)
0620 12:24:10.773007 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.773011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.773495 (+ 484us) server_negotiation.cc:407] Connection header received
0620 12:24:10.773690 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.773693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.773759 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.773807 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.774246 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38746: BlockingRecv error: recv got EOF from 127.0.0.1:38746 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 12:24:10.779165 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.777741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.0.98.61:36335)
0620 12:24:10.777833 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.777837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.778315 (+ 478us) server_negotiation.cc:407] Connection header received
0620 12:24:10.778503 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.778507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.778555 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.778603 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.779072 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38750: BlockingRecv error: recv got EOF from 127.0.0.1:38750 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 12:24:10.784482 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.783063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.0.98.61:36335)
0620 12:24:10.783154 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.783158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.783659 (+ 501us) server_negotiation.cc:407] Connection header received
0620 12:24:10.783832 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.783835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.783879 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.783928 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.784397 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38756: BlockingRecv error: recv got EOF from 127.0.0.1:38756 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 12:24:10.792050 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.790483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.0.98.61:36335)
0620 12:24:10.790605 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.790609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.791160 (+ 551us) server_negotiation.cc:407] Connection header received
0620 12:24:10.791350 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.791355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.791416 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.791467 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.791948 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38758: BlockingRecv error: recv got EOF from 127.0.0.1:38758 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 12:24:10.804288 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.802689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.0.98.61:36335)
0620 12:24:10.802789 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.802794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.803324 (+ 530us) server_negotiation.cc:407] Connection header received
0620 12:24:10.803562 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.803566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.803614 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.803666 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.804187 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38770: BlockingRecv error: recv got EOF from 127.0.0.1:38770 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 12:24:10.824486 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.822733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.0.98.61:36335)
0620 12:24:10.822852 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.822857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.823426 (+ 569us) server_negotiation.cc:407] Connection header received
0620 12:24:10.823627 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.823631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.823682 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.823735 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.824359 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38776: BlockingRecv error: recv got EOF from 127.0.0.1:38776 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 12:24:10.860337 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.859127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.0.98.61:36335)
0620 12:24:10.859256 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.859261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.859409 (+ 148us) server_negotiation.cc:407] Connection header received
0620 12:24:10.859548 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.859552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.859628 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.859752 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.860208 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38784: BlockingRecv error: recv got EOF from 127.0.0.1:38784 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 12:24:10.927809 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:10.926551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.0.98.61:36335)
0620 12:24:10.926725 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:10.926730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:10.926888 (+ 158us) server_negotiation.cc:407] Connection header received
0620 12:24:10.927065 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:10.927069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:10.927126 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:10.927221 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:10.927703 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38792: BlockingRecv error: recv got EOF from 127.0.0.1:38792 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 12:24:11.059473 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:11.058213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.98.61:36335)
0620 12:24:11.058380 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:11.058385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:11.058503 (+ 118us) server_negotiation.cc:407] Connection header received
0620 12:24:11.058697 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:11.058700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:11.058756 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:11.058852 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:11.059314 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38802: BlockingRecv error: recv got EOF from 127.0.0.1:38802 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 12:24:11.319149 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 12:24:11.318011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.0.98.61:36335)
0620 12:24:11.318146 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:11.318150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:11.318264 (+ 114us) server_negotiation.cc:407] Connection header received
0620 12:24:11.318401 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:11.318404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:11.318452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:11.318536 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 12:24:11.319010 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38812: BlockingRecv error: recv got EOF from 127.0.0.1:38812 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 12:24:11.619429 23264 catalog_manager.cc:1259] Loaded cluster ID: d3d4553175e146bf9ecfeaa1060ce982
I20260620 12:24:11.619515 23264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958: loading cluster ID for follower catalog manager: success
I20260620 12:24:11.621184 23264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958: acquiring CA information for follower catalog manager: success
I20260620 12:24:11.622285 23264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55b4bd5165a641d5a554650a8e6eb958: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:24:11.654536 23279 catalog_manager.cc:1259] Loaded cluster ID: d3d4553175e146bf9ecfeaa1060ce982
I20260620 12:24:11.654619 23279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d: loading cluster ID for follower catalog manager: success
I20260620 12:24:11.656299 23279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d: acquiring CA information for follower catalog manager: success
I20260620 12:24:11.657533 23279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d88af46f8fcf47dba89c883089b03b8d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 12:24:11.837893 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:11.833541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.0.98.61:36335)
0620 12:24:11.833797 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:11.833802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:11.833936 (+ 134us) server_negotiation.cc:407] Connection header received
0620 12:24:11.834172 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:11.834175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:11.834298 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:11.834421 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:11.834775 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.835317 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.836290 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.836532 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.836645 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:11.837001 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:11.837472 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:11.837614 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:11.837675 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 12:24:11.837731 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260620 12:24:11.844686 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:11.841156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.0.98.60:42123)
0620 12:24:11.841370 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:11.841374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:11.841388 (+ 14us) server_negotiation.cc:407] Connection header received
0620 12:24:11.841517 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:11.841519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:11.841649 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:11.841759 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:11.841947 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.842450 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.843247 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.843481 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.843574 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:11.843821 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:11.844236 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:11.844333 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:11.844505 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 12:24:11.844569 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260620 12:24:11.852296 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:11.848248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.0.98.60:42123)
0620 12:24:11.848362 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:11.848366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:11.848539 (+ 173us) server_negotiation.cc:407] Connection header received
0620 12:24:11.848707 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:11.848710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:11.848755 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:11.848825 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:11.849210 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.849963 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.851033 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.851354 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.851447 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:11.851822 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:11.851911 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:11.852002 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:11.852092 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 12:24:11.852139 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 12:24:11.852375 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:11.847773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.0.98.61:36335)
0620 12:24:11.847902 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:11.847907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:11.848407 (+ 500us) server_negotiation.cc:407] Connection header received
0620 12:24:11.848552 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:11.848555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:11.848613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:11.848696 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:11.849130 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.849946 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.851108 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.851440 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.851539 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:11.851872 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:11.851945 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:11.852007 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:11.852153 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 12:24:11.852215 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 12:24:11.853060 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 12:24:11.847765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.0.98.62:45693)
0620 12:24:11.848415 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 12:24:11.848420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 12:24:11.848433 (+ 13us) server_negotiation.cc:407] Connection header received
0620 12:24:11.848547 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 12:24:11.848552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 12:24:11.848718 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 12:24:11.848851 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 12:24:11.849189 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.849745 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.851035 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 12:24:11.851312 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 12:24:11.851427 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 12:24:11.851784 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 12:24:11.852127 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 12:24:11.852351 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 12:24:11.852827 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 12:24:11.852887 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":60,"threads_started":1}
I20260620 12:24:11.856052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 23027
I20260620 12:24:11.862462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 23102
I20260620 12:24:11.868221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZkvxte/build/debug/bin/kudu with pid 23176
2026-06-20T12:24:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3062 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6124 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283974 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST